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Abstract 

>*  review  is  provided  of  the  recent  debates  over  whether 
pictorial-like  or  propositional-like  representations  are  most 
appropriate  for  visual  imagery.^  The  arguneit  for  a propositional 
representation  has  largely  taken  tt^  form  of  an  attack  on  the  logical 
coherence  of  pictorial  representatiiOns.  These  attacks  have  not  been 
valid;  one  can  develop  a coherent  dbal-code  model  involving  pictorial 
and  verbal  (non-propositional)  representations.  On  the  other  hand, 
empirical  demonstrations  that  are  \claimed  to  support  pictorial 
representations  fail  to  provide  evidence  that  would  discriminate  such 
representations  from  propositional  ones.  >It  is  argued  that  the  failure 
of  the  anti-pictorial  and  the  pro-pictorial  argunents  stems  from  a 
fOndamental  indeterminancy  in  deciding  issxies  of  representations.  It 
is  shown  that  wide  classes  of  different  representations  can  be  made  to 
yield  identical  behavioral  predictions.  In  particular,  this  potential 
for  mutual  mimicry  holds  between  propositional  and  dual-code 
(pictorial-plus-verbal)  models.  If  one  considers  criteria  such  as 
parsimony  and  efficiency  in  addition  to  prediction  of  behavior,  it  may 
be  possible  to  establish  further  constraints  on  representation.  In 
particular,  it  may  be  possible  to  establish  whether  there  are  two 
codes,  one  for  visual  information  and  one  for  verbal,  or  whether  there 
is  a single  abstract  code.  However,  the  conclusion  of  this  paper  is 
that,  barring  decisive  physiological  data,  it  will  not  be  possible  to 
establish  the  character  of  an  internal  representation — e.g.,  whether  it 
is  pictorial  or  propositional. 
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There  is  no  need  to  docum&it  the  historical  importance  of 
imagery  to  philosophical  and  psychological  discussion  nor  the  important 
place  it  has  in  current  cognitive  psychology.  Ihere  is  also  no  need  to 
review  the  long  debate  over  the  status  of  the  concept  of  imagery. 
Ample  reviews  of  these  matters  exist  in  other  places  (e.g.  Anderson  & 
Bower,  1973;  Bower,  1972;  Bugelski,  1970;  Fodor,  1975;  Kosslyn  & 
Pomerantz,  1976;  Paivio,  1971,  1976;  Pylyshyn,  1973;  Richardson,  1969). 
This  paper  is  concerned  with  analyzing  some  recent  developments  in  the 
debate  over  the  status  of  mental  imagery.  There  are  those  (e.g.  Cooper 
& Shepard,  in  press;  Paivio,  1976;  Kosslyn  & Pomerantz,  1976),  who 
argue  that  visual  imagery  is  encoded  in  terms  of  properties  that  are 
quite  spatial  and  modality  specific.  This  position  is  referred  to  as 
the  "imagery  position."  (This  terminology  will  be  continued  here, 
although  it  is  a potential  source  of  confusion.  The  issue  at  stake  is 
not  whether  imagery  exists,  but  what  mental  representations  underlie 
it.)  On  the  other  side  of  the  issue  are  the  propositional  theorists 
(Anderson  & Bower,  1973;  Chase  & Clark,  1972;  Pylyshyn,  1973i  1976; 
Reed,  1974)  who  argue  that  imagery  is  encoded  in  an  abstract 
propositional  format  and  that  this  same  format  is  used  to  encode  verbal 
information.  In  contrast,  the  imagery  theorists  want  to  make  a sharp 
distinction  between  the  codes  used  for  verbal  and  visual  information. 
The  principal  purpose  of  this  paper  is  to  provide  a critical  analysis 
of  the  argiznents  given  for  both  sides  of  this  debate. 

This  paper  will  have  the  following  structure:  First,  I will 


i 


Mental  Imagery  3 

attempt  to  specify  what  a propositional  theory  is  and  what  an  imagery 
theory  is;  second,  I will  consider  those  argiments  that  have  been  made 
against  an  imagery  position;  third,  I will  consider  those  argunents 
that  have  been  made  for  the  imagery  position;  finally,  I will  take  a 
more  abstract  look  at  these  issues  and  consider  what  evidence  might  be 
useful  in  deciding  between  the  two  positions. 

The  central  thesis  of  this  paper  is  that  arguments  concerning 
the  representation  of  information  have  been  misdirected.  Any  claim  for 
a particular  representation  is  imix>ssible  to  evaluate  unless  one 
specifies  the  processes  that  will  operate  on  this  representation. 
Arguments  for  or  against  a particular  representation  are  only  valid 
assuming  a particular  set  of  processes.  These  arguments  are  not  valid 
assuming  other  processes.  Pylyshyn  (in  press)  has  made  this  point  that 
a theory  must  be  considered  as  a representation  plus  process.  As  he 
writes: 

...the  appropriate  subject  of  our  analysis  of 
representation  should  be  not  the  representation  per 
se  but  a representational  system  consisting  of  the 
pair  (representation , process) . . . 


Despite  this  common  realization,  Pylyshyn 's  conclusions  about  the 
representation  for  imagery  will  be  different  than  mine. 

Defining  Our  Terms 
The  Concept  of  a Proposition 

It  is  classically  accepted  that  there  are  three  features  that 
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define  a proposition  (e.g.,  Frege,  I960).  It  is  abstract;  it  has  a 
truth  value;  and  it  has  rules  of  formation.  Abstractness  is  a concept 
originally  defined  with  respect  to  sentences.  A proposition  can  be 
denoted  by  a sentence  but  is  more  abstract  than  the  sentence.  It  is 
not  tied  to  the  particular  features  of  the  sentence.  Therefore,  the 
attempts  to  equate  "propositional”  with  "verbal"  are  fundanentally 
wrong.  The  notion  of  abstractness  is  related  to  a concept  of 
invariance  under  paraphrase — see  Schank  (1972)  or  Norman  & Rumelhart 
(1975).  This  is  the  idea  that  all  linguistic  paraphrases  or  cross- 
language translations  would  be  assigned  the  same  propositional 
representation.  The  concept  of  invariance  under  paraphrase  is  not  part 
of  the  classical  definition  of  a proposition  and  is  in  dispute  (see 
Anderson,  1976;  Woods,  1975).  Nonetheless,  it  is  the  case  that  all 
propositional  formalisms  practice  some  degree  of  invariance  under 
paraphrase . 

The  notion  of  truth  values  means  that  propositions  are  things 
about  vAiich  it  makes  sense  to  Inquire  whether  they  are  true  or  false. 
Propositions  are  analyzed  as  bearers  of  truth  value  in  logic  from  which 
propositional  representations  were  imported  into  psychology.  However, 
use  of  propositional  theories  does  not  imply  a psychological  theory  of 
meaning  based  on  the  concept  of  truth. 

The  structural  aspect  of  propositions  mean  that  there  are  an 
explicit  set  of  rules  for  determining  uhat  is  a well-formed  proposition 
or  not.  In  logic  rules  of  inference  are  fTaned  with  respect  to  these 
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structural  properties.  In  psychology  cognitive  laws  are  formulated 
with  respect  to  these  structural  properties  (see  Anderson,  1976; 
Anderson  4 Bower,  1973;  Fredericksen,  1975;  Kintsch,  197^;  Norman  4 
Rumelhart,  1975,  for  some  examples  of  cognitive  laws  so  fonnulated). 

Vftiile  there  do  exist  agreed  upon  criteria  for  a propositional 
representation,  these  criteria  lack  the  character  of  operational 
definitions.  Therefore,  it  is  not  a cut  and  dry  matter  to  decide  if  a 
particular  representation  is  propositional.  I will  try  to  follow  what 
has  become  standard  usage  in  the  propositional-imagery  debate.  This 
usage  would  include  as  propositional  representations  such  things  as 
Quillian's  (1969)  associative  network,  Schank's  (1972)  conceptual 
dependency,  or  Winograd's  (1977)  SHRDLU  data  base.  In  each  case  the 
representation  is  abstract,  has  at  least  minimal  rules  of  formation, 
and  the  units  of  representation  can  be  seen  as  having  truth  values.  I 
mean  to  exclude  arrays,  pictures,  S-R  bonds,  unstructured  associations 
such  as  proposed  by  the  British  assoclationists,  procedures,  and  simple 
list  structires.  With  respect  to  the  list  structure  possibility:  Many 
computer  implementations  represent  propositions  as  list  structures.  It 
potentially  could  be  a problem  to  decide  exactly  when  a list  structure 
attains  the  status  of  a proposition,  although  this  ambiguity  does  not 
seem  to  arise  in  practice.  One  possible  criteria  for  propositional 
list  structure  is  that  it  have  clearly  identifiable  predicate  and 
argument  constituents. 
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The  Concept  of  an  Image 

No  one  seems  to  deny  that  there  is  a phenomenon  called  mental 
Imagery.  On  the  other  hand,  there  is  considerable  debate  over  whether 
there  is  a useflil  representational  construct  called  an  image.  Just 
what  an  image  is  thought  to  be,  however,  is  a sore  point.  Generally, 
image  theorists  ascribe  to  images  properties  very  much  like  the 
properties  that  are  given  in  phenomenological  reports  by  subjects  while 
experiencing  a visual  image.  Thus,  these  theorists'  use  of  the  term 
"image"  might  be  construed  as  an  attempt  by  them  to  make 
phenomenological  features  the  primitives  of  the  psychological  mechanism 
that  accounts  for  these  phenomenological  reports  and  for  other  relevant 
data  on  use  of  imag«*y.  In  contrast,  propositional  theories  must 
attempt  to  derive  such  imagery  reports  fhom  mechanisms  which  do  not 
have  much  in  common  with  the  phenomenological  reports. 

The  paper  by  Pylyshyn  (1973)  was  Important  in  highlighting  a 
nunber  of  issues  concerning  the  role  of  an  image  as  an  explanatory 
concept.  He  argued  that  the  underlying  notion  of  an  image  is  built  on 
the  metaphor  of  a picture: 

The  whole  vocabulary  of  Imagery  uses  a 
language  appropriate  for  describing  pictures  and 
cne  process  of  perceiving  pictures.  We  speak  of 
clarity  and  vividness  of  images,  of  scanning 
Images,  of  seeing  new  patterns  in  images,  and  of 
nmaing  objects  or  properties  depicted  in  images. 

(p.8) 

Pylyshyn  and  Anderson  & Bovier  have  criticized  this  picture-metaphor 
model  of  an  image  on  many  counts  including  that  it  makes  predictions 
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which  have  been  empirically  disconfinned , that  there  are  internal 
inconsistencies  in  the  use  of  this  model,  that  there  are  better  models 
for  many  phenomena,  and  that  the  model  involves  many  inelegant, 
unpalatable  assumptions. 

However,  Kosslyn  and  Pomerantz  (1977)  and  Paivio  (1976)  have 
argued  that  the  picture-metaphor  is  a straw  man,  that  few  imagery 
theorists  support  such  a metaphor  in  any  literal  sense.  They  concede 
that  the  picture-metaphor  is  not  defensible.  As  Paivio  writes: 


The  wax  tablet  of  picture  metaphor  is  open  to 
the  kinds  of  criticisms  that  were  directed  at  it 
over  the  ages.  Today,  however,  they  are  largely 
directed  at  a straw  man  because  no  imagery 
researcher  accepts  the  metaphorical  view  as  a 
working  theory,  (p.2) 


Kosslyn  & Pomerantz  write: 


Pylyshyn's  attacks  are  based  on  a particular 
definition  of  imagery,  namely  the  picture- in  the- 
head  hypothesis.  We  agree  with  Pylyshyn  that  this 
approach  is  untenable  but  fail  to  see  what  is 
g^ned  by  attacking  such  a stravman.  No  serious 
student  of  imagery  holds  this  view,  (p.57) 


The  problem  is  that  imagery  theorists  have  failed  to  say 
precisely  what  an  image  is.  It  is  somev^iat  hard  to  debate  about  the 
status  of  an  object  vAien  no  one  will  specify  what  the  object  is.  Both 
Paivio  and  Kosslyn  & Pomerantz  propose  that  an  image  of  an  object  may 
be  equated  to  the  product  of  perception  of  the  object,  and  they 
distinguish  this  representation  from  a picture.  However,  the  current 
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state  of  knowledge  about  perception  is  inadequate  to  decide  the  : 

appropriate  representation  for  an  image.  Also  it  is  the  case  that  the 
propositional  theorists  (e.g.  Anderson  & Bower,  1973;  Palmer,  1975; 

Pylyshyn,  1973)  want  to  equate  the  output  of  perception  with  a 
propositional  representation.  So  they  too  endorse  the  claim  that 
imagery  and  perception  have  the  same  representation. 

Consider  some  of  the  other  explications  of  an  image  offered  by 
Kosslyn  and  Pomo'antz: 

...images  are  like  surface  displays  generated 
on  a cathode  ray  tii>e  by  a computer...  (p.70) 


Images,  once  formed  are  wholes  that  may  be 
compared  to  percepts  in  a template-like  manner. 

(p.66) 


What  is  a display  on  a cathode-ray  tube  or  a template  but  another  name 
fbr  a spatial  array  of  light  information?  What  is  a spatial  array  of 
light  Information  but  a pictire?  It  seems  that  the  picture-metaphor  is 
the  only  available  model  of  im^ery.  By  "picture”  I mean  some  format 
that  represents  information  as  a spatially  structured  array  of  light 
information.  Consider  the  recent  model  by  Kosslyn  4 Shwartz  (1977). 
This  computer  simulation  model  is  clearly  the  most  owicrete  theory  by 
anyone  in  the  imagery  canp.  In  this  model,  an  image  is  represented  by 
a set  of  points  encoding  the  location  in  two-dimensional  space  of  the 


contour  points  of  an  imaged  object.  Clearly,  such  a representation  is 
"pictorial”  in  my  sense. 
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Thus,  It  seems  the  picture  metaphor  is  the  imagery  theory. 
Moreover,  I think  all  sides  have  been  premature  in  abandoning  the 
picture  metaphor.  Vi/hat  I would  like  to  do  is  to  consider  a semi- 
explicit picture  model  and  show  that  it  is  not  susceptible  to  the 
criticisms  typically  made  of  such  a model. 

What  I want  to  defend  is  not  simply  a picture  model,  but  that 
model  as  a part  of  a larger  theory.  This  larger  theory  is  basically 
the  dual  code  model  of  Paivio  (1971).  He  proposed  that  there  are  two 
types  of  information  representations,  visual  and  verbal.  The  visual 
representations  I will  equate  with  pictures.  The  verbal  information 
will  be  equated  with  word  strings.  These  words  strings  can  be  thought 
of  as  acoustic  objects  temporally  structured,  although  this 
interpretation  is  not  essential  to  what  follows.  What  is  important  is 
that  they  not  be  interpreted  as  abstract  propositions.  I think  it  is  a 
correct  reading  both  of  Paivio 's  exposition  of  his  theory  and  of  the 
attacks  on  it  by  the  propositional  canp  that  nothing  in  his  theory  can 
be  considered  abstract  propositional.  It  is  important  that  the  theory 
I propose  be  faithful  to  the  Paivio  position.  If  it  is,  it  definitely 
should  be  clear  that  I have  not  bolstered  the  power  of  a picture  theory 
by  sneaking  in  propositions. 

A Verbal-Pictorial  Dual  Code  Theory 


The  picture  model  I will  discuss  is  also  somewhat  similar  to 
the  idea  of  Kosslyn  and  Shwartz.  The  principal  difference  is  that  my 
model  is  motivated  purely  to  provide  a counter-example  to  anti-imagery 
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claims.  Therefore,  it  contains  some  "outrageous**  suggestions  which  1 
would  probably  want  to  modify  were  I to  advance  this  model  as  a serious 
proposal.  The  point  of  the  model  is  not  its  plausibility,  but  rather 
how  a very  *'brazen*'  picture  model  cannot  be  disconfirmed . If  this 
model  is  not  subject  to  any  difficulties,  presun^ly  models  like  that 
of  Kosslyn  & Shwartz,  constructed  with  more  concern  for  the  prevailing 
sensitivities  of  the  cognitive  camunity,  will  also  not  be  subject  to 
decisive  contradiction. 

Images  will  be  interpreted  as  m x n arrays  of  dots  where  each 
dot  can  be  specified  according  to  color  and  intensity.  We  will  also 
have  as  a different  storage  medivxn  (principally  for  verbal  information) 
strings  of  words.  There  will  also  be  directed  associations  that  can 
connect  strings  and  pictures. 

Consider  how  we  might  represent  the  chessboard  of  an  end  game 
position  in  Figure  1.  Figure  2 shows  a possible  representation.  Note 
that  the  situation  has  been  broken  into  a nunber  of  overlapping  images 
connected  by  associations  (arrows).  This  illustrates  that  a single 
external  situation  may  be  represented  by  a nunber  of  images.  A 
possible  motivation  for  this  fragmentation  is  the  fact  that  there  may 
be  capacity  limitations  on  the  amount  of  information  that  can  be  held 
in  a single  image.  Note  also  that  there  are  portions  of  the  chessboard 
not  represented.  This  illustrates  the  fact  that  images  need  not  be 
complete.  Note  also  that  the  black  king  is  inaccurately  located.  This 
illustrates  the  fact  that  impginal  representations  need  not  be 
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veridical.  Note  also  the  verbal  strings  like  "knight  attacks  king"  are 
tagged  to  the  images.  This  is  one  way  of  providing  meaningful 
interpretations  to  the  image. 


Insert  Figures  1 and  2 about  here 


A final  feature  to  note  of  this  representation  is  that  the 

picture  unit  tends  to  segment  out  meaningful,  coherent  subunits  of  the 
2 

picture.  This  segmentation  reflects  the  operation  of  perceptual 
procedures  which  encode  and  Interpret  the  original  object.  One  can 
think  of  these  as  perceptual  chuiks  such  as  discussed  by  Chase  & Simon 
(1973)>  So,  as  the  imagery  theorists  would  postulate,  this 
representation  is  identified  with  the  output  of  the  perceptual  routine. 

The  representation  in  Figure  2 seems  a faithful  embodiment  of 
the  dual  code  theory  proposed  by  Paivio  (1971).  That  is,  there  is  a 
pictorial  code,  a verbal  code,  and  connections  between  the  two.  Armed 
with  this  dual  code  model  let  us  consider  some  of  the  criticisms  that 
are  made  of  images  and  of  their  role  in  dual  code  theories. 


Criticisms  of  Picture  Theory 
Images  are  Pre- interpreted 

Oie  f1*equent  critician  made  of  imagery  theories  is  that  images 
behave  as  if  they  were  interpreted  whereas  pictures  (Just  arrays  of 
light  information)  cannot: 

...one's  representation  of  a scene  must 
contain  already  differentiated  and  interpreted 
perceptual  aspects.  In  other  words,  the 
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representation  is  far  from  being  raw,  and  so  to 
speak,  in  need  of  'perceptual'  interpretation. 
(Pylyshyn,  1973,  p.10) 


We  would  argue  that  people  store  perceptual 
Interpretations  of  scenes  rather  than  'raw, 
unanalyzed,  textured  details'  of  such  scenes. 

(Anderson  4 Bower,  1973,  p.^53). 

Pylyshyn  points  out  some  of  the  many  ways  in  which  our  memory  for  an 
image  behaves  as  if  it  were  interpreted.  We  can  Imnediately  retrieve 
an  image  directly  from  a meaningful  description  like  "my  living  room." 
This  is  not  to  be  expected  if  we  had  to  scan  through  a series  of 
pictures.  However,  it  is  a feature  of  the  representation  in  Figure  2 
that  it  can  be  accessed  via  verbal  labels.  The  associative 
interconnections  between  strings  and  pictures  basically  provide  this 
conceptual  indexing. 

Pylyshyn  also  notes  that  we  can  access  meaningful  parts  of  an 
image  such  ' s "the  light  fixture  in  my  dining  room"  and  that  when  we 
forget  part  of  an  image  we  tend  to  lose  a meaningful  part  and  not  a 
physical  portion  of  the  picture  that  might  cross  many  meaningful  units. 
However,  the  representation  in  Figure  2 is  organized  into  meaningful 
units — producing  access  to  meaningful  parts  and  a potential  for  loss  of 
meaningful  parts. 

It  is  argued  that  it  is  wasteful  to  perceptually  analyze  a 
scene  but  to  store  only  the  raw  scene  rather  than  the  analysis. 
scene  analysis  will  have  to  be  performed  again  when  the  scene  is 
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retrieved.  There  are  two  remarks  that  can  be  made  about  this 
objection.  First,  perceptual  interpretation  does  not  seem  very  costly 
for  the  hvxnan  system.  So  it  is  relatively  insignificant  whether  it  has 
to  be  perfonned  again.  Second,  there  is  no  reason  why  the  output  of 
the  perceptual  process  might  not  be  a representation  like  that  in 
Figure  2 — that  is,  segmented  into  subpictures  with  relations  indicated 
among  these  subpictures,  but  the  structure  of  the  subpictures  not 
analyzed. 

It  is  clear  from  considering  these  objections  that  the 
representation  in  Figure  2 is  "interpreted."  This  interpretation  is 
provided  by  the  network  of  associations.  So,  there  is  nothing 
Incompatible  between  the  concepts  of  a picture  memory  and  an 
interpreted  memory.*' 

Capacity  Limitations 

It  is  argued  that  the  storage  demands  to  encode  these  pictures 
vfould  be  enormous  and  exceed  the  capacity  of  the  brain.  However,  given 
current  ignorance  about  the  capacity  of  the  brain  this  remark  has 
virtually  no  force.  It  is  also  possible  in  the  dual  code  model  of 
Figure  2 to  reduce  the  storage  demands  by  choosing  a crude  grid  or  by 
incompletely  representing  the  objects.  It  is  also  worth  considering 
the  storage  demands  of  alternatives  to  a picture  model.  For  instance, 
consider  the  Baylor  (1971)  model  advocated  by  Pylyshyn.  This  model 
requires  separately  representing  every  vertex,  line,  surface,  and 
object  (and  some  of  these  redundantly)  in  Baylor's  I>space  and  S-space. 


Mental  Imagery  14 


The  demands  of  such  a representation  also  seem  very  large.  The  simple 
fact  may  be  that  there  is  a great  deal  of  information  in  an  image  and 
any  representation  of  an  image  will  have  to  acknowledge  this  fact. 
Representation  of  Vagueness 

Information  can  be  represented  vaguely  in  a picture,  but  it  is 
claimed  that  the  character  of  the  vagueness  does  not  match  the 
character  of  vagueness  in  introspective  reports  of  images.  A picture 
can  be  of  poor  resolution  but  the  vagueness  of  himan  images  seems 
Inexplicable  in  terms  of  simple  resolution.  For  instance,  Pylyshyn 
points  out  that  we  may  be  able  to  recall  what  objects  were  in  the  room 
without  recalling  their  exact  spatial  location.  A poor  resolution 
picture  of  the  room  is  more  likely  to  preserve  relative  spatial 
information  than  to  preserve  object  identity. 

However,  this  is  not  an  Insurmountable  problem  with  the  picture 
metaphor.  Consider  again  the  representation  in  Figure  2:  It  is 
possible  to  have  two  subpictures  of  the  chessboard  sections  without 
encoding  their  relative  spatial  location  or  with  only  encoding  the 
relation  as  "beside,”  not  specifying  whether  to  the  right  or  left. 

Anderson  & Bower  (1973)  also  wondered  how  it  is  that  one  could 
remember  that  a lamp  was  in  the  room  without  being  able  to  recall  any 
details  about  the  appearance  of  the  lamp.  This  phenomenon  has  multiple 
possible  explanations  within  a dual  code  model.  It  is  possible  that  we 
have  stored  the  verbal  label  "lamp"  rather  than  the  picture  in  our 
scene  description.  Another  possibility  is  that  we  have  stored  a 
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picture  but  have  also  tagged  that  picture  with  "incertain”  or  some 
other  label  to  Indicate  we  should  not  take  the  visual  details 
seriously.  This  might  be  done  if  the  transmission  of  information  ft-om 
the  scene  to  the  laop  encoding  was  not  accurate.  Yet  another 
possibility  is  that  we  have  replaced  the  actual  lamp  with  a picture  of 
a prototypical  lamp.  Recognizing  that  this  is  a prototypical  lamp,  the 
system  is  not  willing  to  use  this  as  a basis  for  reporting  details 
about  the  particular  lamp  we  saw.  It  might  be  more  efficient  to  use  a 
prototypical  lamp  than  the  actual  lamp  if  it  was  easier  to  transmit 
information  from  the  internal  representation  of  a prototype  than  from 
an  external  scene.  The  possibility  of  an  uncertain  lamp  or 
prototypical  lamp  illustrates  an  important  possibility  that  has  not 
been  frequently  recognized  in  the  debate  over  imagery.  This 
possibility  is  that  there  is  internal  information  (i.e.,  the  uncertain 
lamp  or  prototype)  which  subjects  cannot  report. 

Metaphors  are  Misleading 

Another  criticism  that  Pylyshyn  levels  against  the  picture 
metaphor  is  that  it  is  misleading,  that  it  causes  people  to  postulate 
mental  processes  from  analogy  to  operations  we  can  carry  out  on 
pictures : 


For  example,  one  misleading  implication 
involved  in  using  the  imagery  metaphor  is  that  what 
we  retrieve  from  memory  vAien  we  image,  like  what  we 
receive  from  our  sensory  systems,  is  some  sort  of 
undifferentiated  (or  at  least  not  fully 
Interpreted)  signal  or  pattern,  a major  part  of 
which  (although  perhaps  not  all)  is  simultaneously 
available.  This  pattern  is  subsequently  scanned 
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perceptually  in  order  to  obtain  neaningAjl 
information  regarding  the  presence  of  objects, 
attributes,  relations,  etc.  (p.8) 

However,  it  is  an  open  question  whether  the  metaphor  is  misleading.  In 
particular,  results  such  as  those  of  Kosslyn  (1973)  on  scanning  or 
Shepard  (1975)  on  mental  rotation  of  images  would  seem  to  imply  the 
metaphor  leads  one  in  quite  fruitful  paths. 

Semantic  Effects  in  Picture  Memory 

Anderson  & Bower  made  much  of  the  evidence  that  pictures  are 
remembered  better  when  they  can  be  Interpreted.  DeGroot  (1965)  finds 
better  memory  for  meaningful  chess  positions.  Wiseman  & Neisser  (1971) 
find  better  memory  for  Mooney  pictures  tnat  subjects  man2^e  to 
meaningAjlly  interpret.  Bower  & Karlin  (197^)  report  similar  results. 
Goldstein  and  Chance  (1970)  have  shown  poorer  memory  for  snow  flakes, 
which  permit  little  meaningful  structuring,  than  for  faces  which  do. 
These  results  are  certainly  contrary  to  claims  (e.g.  Bugelski,  1971) 
about  the  mnemonic  superiority  of  the  image  system,  but  they  really  say 
little  about  whether  pictures  are  used  as  a mediun  of  storage.  It  Is 
quite  possible  that  pure  pictures  are  a poor  storage  mediun.  It  may  be 
that  some  meaningful  structure  as  in  the  chessboard  of  Figure  2 is 
needed  for  good  memory.  Perhaps  retrieval  schemes  require  verbal 
labels  to  provide  access  routes. 
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Staining  Up  the  Picture  Metaphor 

In  conclusion,  it  has  not  been  shown  that  there  is  anything 
incoherent,  contradictory,  or  impractical  in  using  pictures  as  a 
representational  format.  Whether  pictures  are  suitable  depends  on  the 
processes  that  one  assunes  operate  on  them.  To  be  sure,  one  can  assune 
inadequate  processes  for  a picture  representation  (e.g.,  a forgetting 
process  that  consisted  of  fading  rather  than  loss  of  meaningful 
subparts)  as  did  Anderson  & Bower,  Pylyshyn,  and  even  some  of  the  image 
theorists.  But,  then  again,  one  need  not. 

Propositional  Representations 

Besides  the  supposed  empirical  and  logical  inadequacies  of 
picture  theories  an  independent  line  of  argument  against  them  is  that 
picture  representations  are  unnecessary.  This  argiment  has  two 
subclaims.  First,  it  is  argued  that  a propositional  representation  is 
essential  on  independent  grounds.  Second,  it  is  argued  that  a 
propositional  representation  can  handle  all  the  phenomena  ascribed  to  a 
pictorial  representation.  Therefore,  on  grounds  of  parsimony  it  is 
unnecessary  to  assune  pictorial  representations  in  addition  to 
propositional  representations.  This  argunent  would  have  some  validity 
if  both  sUbclaims  were  true.  However,  the  first  subclaim  has  not  been 
established. 

Are  Propositional  Representations  Necessary? 


Pylyshyn  argues  that  it  is  necessary  to  propose  a propositional 
code  to  explain  how  it  is  that  people  can  describe  pictures  in  words  or 
create  pictures  to  illustrate  verbal  material.  Similar  proposals  for 
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an  interlingua  have  been  made  by  Clark  & Chase  (1972),  by  Anderson  & 
Bower  (1973)  and  by  Fodor  (1975).  The  abstract  propositional  code 
would  serve  as  a neutral  format  into  which  and  out  of  which  pictorial 
■Id  verbal  information  could  be  translated.  It  serves  as  a "half-way 
house"  for  the  process  of  translating  between  the  two  peripheral  codes. 
As  Pylyshyn  (1973)  writes: 


. . .the  need  to  postulate  a more  abstract 
representation — one  which  resembles  neither 
pictires  nor  words  and  is  not  accessible  to 
subjective  experience — is  unavoidable.  As  long  as 
we  recognize  that  people  can  go  from  mental 
pictures  to  mental  words  or  vice  v«*sa,  we  are 
forced  to  conclude  that  there  must  be  a 
representation  (which  is  more  abstract  and  not 
avails^le  to  conscious  experience)  which 
encompasses  both.  There  must,  in  other  words,  be 
some  common  format  or  interlingua...  (p.5). 


However,  this  argunent  is  flawed  with  a serious  internal  inconsistency. 
It  is  argued  that  to  translate  from  code  1 to  code  2 it  is  necessary  to 
translate  code  1 into  a new  code,  code  3,  and  from  code  3 to  code  2. 
However,  if  true,  this  would  lead  to  an  infinite  regress.  To  translate 
from  code  1 to  code  3t  a new  code  4 would  be  needed  and  so  on.  It  is 
simply  not  the  case  that  it  is  necessary  to  have  a propositional  or  any 
other  intermediate  code  for  translation.  By  careful  analysis,  it  might 
be  possible  to  show  that  an  interlingua  makes  the  translation  process 
more  efficient,  but  such  an  analysis  has  not  been  forthcoming. 

Another  argunent  used  by  both  Anderson  4 Bower  and  by  Pylyshyn 
is  that  a propnsltional  code  is  needed  to  represent  meaning.  For 
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instance,  Pylyshyn  argues  that  one  needs  this  code  to  represent  a 
relation  like  "attacked-by**  in  a chessboard.  His  example  inspired 
Figure  2 vghich  shows  this  information  can  be  represented  by  verbal 
strings.  The  "meaning"  can  be  contained  in  the  processes  that 
interpret  the  strings. 

Anderson  & Bower  argue  that  a propositional  code  is  needed  to 
deal  with  the  considerable  evidence  that  subjects  tend  to  remember  only 
the  meaning  of  sentences  and  not  their  exact  wording.  So,  for 
instance,  subjects  show  confusion  about  whether  they  heard  an  active 
sentence  or  the  passive  equivalent  (Sachs,  1967).  There  are  any  number 
of  explanations  of  the  phenomena  which  do  not  require  postulations  of  a 
propositional  code.  First,  one  could  propose  that  subjects  represent 
these  sentences  in  memory  by  pictures  (e.g.  a picture  of  a boy  hitting 
a girl) . Such  pictures  would  represent  the  meaning  but  not  the 
original  form.  Another  possibility  is  that  subjects  convert  various 
sentences  to  a coomon  string  format.  So  all  sentences  might  be 
represented  as  active.  The  processor,  knowing  this,  would  be  smart 
enough  not  to  use  the  form  of  the  string  in  memory  as  a basis  for 
judging  the  form  of  the  string  that  was  studied.  Another  possibility 
is  that  the  exact  string  heard  is  stored  in  memory  but  the  processor 
does  not  pay  attention  to  the  difference  between  active  and  passive 
sentences  because  normally  this  is  not  important. 
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Power  of  a Propositional  Code 

It  seems  to  be  a generally  accepted  claim  that  any  well- 
specified  set  of  information  can  be  represented  by  a set  of 

4 

propositions.  Thus,  propositions  can  represent  any  information  in  an 
image  or  in  a sentence  or  information  from  any  other  source.  One  might 
argue  fhom  this  fact  that  propositional  representations  are  appropriate 
for  rejM'esenting  images.  To  my  knowledge,  no  one  has  explicitly  made 
this  argunent  although  Kosslyn  and  Pomerantz  ascribe  it  to  Pylyshyn. 
The  counter  argunent  is  obvious:  Even  if  all  Information  can  be 
represented  propositionally,  it  does  not  follow  that  the  propositional 
representation  will  lead  to  the  correct  empirical  predictions. 

The  power  of  propositional  representations  has  been  used  by 
Kosslyn  and  Pomerantz  to  argue  against  them: 


/t 


The  problem  with  propositional  theories,  on 
the  other  hand,  is  that  they  may  be  too  powerful. 
They  possess  no  inherent  constraints,  and  the 
theorist  must  add  restrictions  onto  his  theory  to 
make  it  conform  with  experimental  observations. 
The  propositional  language  is  so  powerful  that  one 
can  use  it  to  formulate  almost  any  kind  of  theory 
that  one  desires,  predicting  with  equal  ease,  it 
would  seem,  any  experimental  finding  or  its 
converse,  (p.62) 


I will  return  to  this  question  of  the  plasticity  of  representations 
later  in  this  paper  to  give  it  a more  thorough  examination.  However, 
at  this  point  a specific  rejoinder  is  required  to  this  criticism:  A 
representation  without  any  process  assunptions  is  not  a theory.  By 
making  different  process  assunptions  it  is  possible  to  have  quite 
different  theories  with  the  same  propositional  representation.  We 
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cannot  test  representations  but  only  representation-process  pairs.  It 
is  not  an  argunent  against  a representation-process  theory  that  there 
ha^^ns  to  be  a different  theory  with  the  same  representation  but 
different  process  that  makes  the  opposite  prediction. 

Propositional  Representations  and  Inference-Making 

It  is  sometimes  assuned  (e.g.,  Anderson,  1976;  Anderson  & 
Kline,  1977;  Fodor,  1975;  Pylyshyn,  1973)  that  there  is  a particular 
affinity  between  propositional  representations  and  the  task  of  making 
inferences.  The  claim  is  that  propositional  representations  are 
particularly  well-suited  for  the  operation  of  inference  making. 
Propositions,  because  of  their  abstract  truth-bearing  character  only 
represent  what  is  necessary  to  judge  the  validity  (or  plausibility)  of 
an  inference.  There  can  be,  therefore,  a reduction  in  the  complexity 
and/or  nunber  of  inference  rules  if  they  are  formulated  in  terms  of 
propositions.  For  instance,  most  propositional  systems  would  represent 
active  and  passive  sentences  identically.  Anything  that  follows  from 
an  active  also  follows  from  a passive.  A propositional  system  avoids 
the  need  to  have  separate  inference  rules  for  actives  and  passives. 
This  is  probably  the  reason  why  almost  every  artificial  intelligence 
program  fbr  making  inferences  uses  some  variety  of  a propositional  data 
base. 

However,  these  considerations  do  not  force  the  conclusion  that 
human  inference  making  is  done  in  terms  of  abstract  propositions.  It 
may  be  the  case  that  while  artificial  intelligence  programs  are  capable 
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of  such  abstraction,  humans  are  not.  In  fact,  there  is  some  evidence 
that  the  hunan  system  is  not  capable  of  such  abstraction:  It  has  been 
shown  that  semantically-irrelevant  linguistic  details  of  an  argunent 
affect  the  time  and  success  of  making  a valid  conclusion  (e.g.  Clark, 
1969;  Rips  4 Marcus,  1976).  While  these  data  can  be  interpreted  within 
a propositional  model,  they  do  serve  to  deflate  claims  about  the 
advantage  of  propositional  representations  in  abstracting  out  only 
truth-relevant  information  for  inference  making. 

Pro-Imagery  Argunents 

So  far,  we  have  considered  arguments  against  imagery  theories 
and  arguneits  for  propositional  representations.  These  two  types  of 
argunents  define  the  "anti- imagery"  position.  Now  I would  like  to  turn 
to  the  consideration  of  the  arguments  that  have  been  made  for  imagery 
theories.  By  "imagery  theories,"  I mean  theories  based  on  the  picture 
metaphor.  As  noted  earlier,  this  is  the  only  explicit  interpretation 
available  for  the  concept  of  an  image. 

Computational  Advantages  of  Imagery  Representations 

Kosslyn  and  Pomerantz  argue  that,  while  the  same  information 
can  be  represen oed  via  propositions  or  images,  it  is  easier  to  perform 
certain  types  of  computations  on  images.  Therefore,  images  have  a 
computationally  useful  function.  It  is  further  assuned  that,  if 
something  is  useful,  it  is  likely  to  be  found  in  the  hunan  head.  I 
will  be  making  a similar  argunent  for  efficiency  considerations  later 
In  this  paper.  A basic  problem  is  that  claims  about  one  representation 
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being  more  efficient  than  another  are  always  based  on  implicit 
assumptions  about  the  processes  to  be  used  with  the  two 
representations.  With  other  process  assunptions  these  claims  may  no 
longer  be  valid. 

Kosslyn  & Pomerantz  present  an  example  to  make  their  point. 
They  contrast  two  formats  for  representing  geographical  information;  a 
map  with  cities  on  it  versus  a chart  of  intercity  distances.  The  first 
is  thought  of  as  similar  to  an  image  representation  and  the  second  as 
more  similar  to  a propositional  representation.  They  claim  that  these 
are  "isomorphic"  to  each  other  in  all  important  respects  since  they 
contain  identical  information  and  either  one  can  be  generated  from  the 
other  {p.60).  (This  is  not  totally  true  in  that  north-south 
orientations  cannot  be  recovered  from  the  chart.)  They  claim  that 
different  types  of  computations  are  appropriate  to  the  two 
representations : 

If  we  want  to  know  quickly  whether  three 
cities  fall  on  a straight  line,  we  consult  a map; 
if  we  want  to  know  the  total  distance  of  an  air 
flight  from  New  York  to  Los  Angeles  to  Miami  we 
consult  the  chart.  (p.61) 

However,  these  claims  are  only  true  assixning  a particular  algorithm  on 
a particular  device.  There  is  a relatively  simple  algorithm  for 
solving  the  first  problem  given  chart  distances:  See  if  two  of  the 
distances  sun  to  the  third.  In  some  implementations,  this  algorithm 
would  be  more  efficient  than  any  map  scanning  operation.  Conversely, 
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if  it  were  difficult  to  add  the  distances,  but  one  had  a string  marked 
off  so  as  to  translate  string  length  into  miles,  the  second  problem 
might  be  better  solved  with  the  map. 

So  it  is  not  possible  to  decide  issues  of  efficiency  of  a 
representation  without  knowledge  of  the  procedures  that  will  be 
operating  on  them.  However,  later  I will  be  arguing  a point  somewhat 
similar  to  Kosslyn  & Pomerantz  which  is  that  efficiency  considerations  | 

may  lead  to  the  need  for  specialized  representations.  In  contrast  to  ! 

Kosslyn  & Ponerantz,  I do  not  think  we  can  decide  on  the  nature  of  the 
specialized  representations. 

Argunents  Based  on  Introspection 

Introspections  probably  provide  the  most  intuitively  compelling 
evidence  for  the  existence  of  a pictuire-llke  image  code  and  for  the 
distinction  between  this  code  and  a verbal  code.  Many  people  find 
Introspection  about  the  picture-like  quality  of  imagery  extremely 
striking.  Of  course,  there  are  those  who  report  little  or  no  such 
introspective  experience.  Informal  count  seems  to  indicate  that  these 
non-picture-imagers  tend  to  be  the  doubters  of  imagery  theory.  As 
scientists,  however,  we  must  go  beyond  the  intuitive  force  of  our 

1 

internal  experiences  and  ask  what  is  the  logical  force  of  the  presence  ’ 

i 

or  absence  of  such  introspective  reports.  The  introspective  reports 

are  data  that  require  explanations  like  any  other  data.  However,  there  I 

is  no  reason  to  suppose  that  the  best  representation  to  account  for 

verbal  reports  of  picture-like  properties  of  an  image  is  a picture.^  A 
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computer  program  could  be  written  to  deliver  such  reports  fVam  a 
propositional  data  base  (for  instance,  see  Moran,  1973).  The  tendency 
to  assuae  a picture  representation  to  account  for  reports  of  pictures 
illustrates  a wide-spread  fallacy  in  discussions  of  representations 
that  the  best  way  to  explain  data  with  property  P is  to  assume  a 
representation  with  property  P. 

Arguments  for  Imagery  From  Experimental  Demonstration 
Host  imagery  theorists  do  not  base  their  position  on  such 
introspective  evidence — at  least  not  publically.  Rather,  they  will 
marshall  a large  series  of  experiments  to  support  their  position. 
Therefore,  it  is  important  to  consider  some  of  the  classes  of  data 
presented  and  to  evaluate  the  ability  of  these  to  decide  issues  of 
Internal  representation.  An  early  class  of  data  concerned  the  supposed 
superior  mnemonic  capacity  of  the  imagery  system — better  memory  for 
pictures  or  for  imaginable  words,  better  memory  under  instructions  to 
image,  etc.  The  culmination  of  the  line  of  evidence  is  Paivio's  (1971) 
book  which  provides  an  extensive  review.  It  appears  that  reliance  on 
such  data  is  waning.  One  reason  for  this  waning  is  that  further 
evidence  has  indicated  that  the  pictorial  material  is  not  superior  to 
verbal  material.  First  of  all,  verbal  material  when  "deeply  processed" 
can  display  similarly  high  levels  of  memory  (Anderson  & 8ower,  1973; 
Anderson,  1976;  see  Cralk  & Loctdiart,  1972  for  a general  discussion  of 
levels  of  encoding).  Second,  the  good  memory  for  pictorial  material 
only  seems  to  apply  when  that  pictorial  material  can  be  meaningfully 


Mental  Imagery  26 


interpreted  (Anderson  & Bower,  1973;  Bower  & Karlin,  1974;  Goldstein  & 
Chance,  1971;  Mooney,  1959;  Wiseman  & Neisser,  1971). 

There  now  appears  to  be  a second  generation  of  research  to 
support  the  imagery  approach.  This  research  is  more  concerned  with 
operations  on  images  in  immediate  memory  rather  than  the  treatment  of 
images  in  long-term  memory.  The  logic  of  this  research  has  been 
succinctly  analyzed  by  Shepard  and  Podgorny  (in  press).  The  attempt  is 
to  show  that  when  subjects  process  objects  mentally  the  functional 
consequences  for  behavior  are  quite  similar  to  those  when  subjects 
process  the  actual  objects.  This  is  evidence  that  the  mental  or 
Imaginal  representation  of  an  object  is  the  sane  as  the  perceptual 
representation.  Shepard  and  Podgorny  note  that  this  equivalence  need 
not  imply  that  either  representation  is  picture-like.  However,  the 
fhequent  interpretation  of  such  an  equivalence  is  that  the  imaginal 
representation  must  be  picture-like  because  (assunption)  the  perceptual 
representation  is  picture-like.  At  this  level,  the  logic  of  these 
experiments  Is  not  at  all  compelling  because  many  theorists  (Anderson  & 
Bower,  1973;  Palmer,  1975;  Pylyshyn,  1973;  Winston,  1970)  would  reject 
the  notion  of  a picture-like  representation  for  the  products  of 
perception  and  rather  assert  a propositional  representation.  In  this 
section  I would  like  to  focus  on  some  of  the  specific  demonstrations 
and  see  if  they  have  any  additional  force  beyond  this  general  argixnent. 
I will  not  have  room  to  consider  all  types  of  data  that  have  been 
enumerated  in  such  papers  as  Kosslyn  & Pomerantz  (1977),  Paivio  (1975), 
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and  Shepard  & Podgorny  (in  press).  However,  I will  consider  a 
representative  subset.  This  will  be  enough  to  make  a general  point. 

Mental  Rotations 

One  of  the  most  influential  phenomena  uncovered  in  recent 
research  in  cognitive  psychology  has  been  that  of  mental  rotation  (see 
Metzler  & Shepard,  1974;  Cooper  & Shepard,  in  press).  The  basic 
finding  is  that  the  time  to  decide  that  one  object  is  a rotation  of 
another  object  is  a monotonic,  and  often  linear,  function  of  the  anount 
(degrees)  of  rotation.  This  is  taken  as  evidence  that  subjects 
OMntally  rotate  an  image  of  one  object  of  the  pair  into  congruence  with 
the  other  object.  The  continuous  nature  of  the  function  is  taken  as 
evidence  that  the  subject  must  go  through  all  or  many  intermediate 
states  in  rotating  the  object.  In  the  Kosslyn  & Shwartz  simulation, 
the  image  is  moved  through  a series  of  small  changes  in  orientation. 

It  is  a simple  matter  to  propose  a propositional  model  which 
mimics  this  image  model.  The  model  would  Involve  a propositional 
description  of  an  object  and  of  its  orientation  in  space.  Just  as 
Kosslyn  & Shwartz  compute  a series  of  small  changes  in  their  image  so  a 
series  of  small  changes  can  be  computed  in  the  propositional 
representation.  Given  that  it  is  so  easy  to  produce  the  phenomenon  in 
a propositional  model , one  might  wonder  why  propositional  theorists 
(e.g.  Anderson  & Bower,  1973)  have  been  so  loathe  to  propose  it  and  why 
most  find  the  image  account  more  appealing  than  the  propositional 
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•eeouit.  Consider  Kosslyn  & Ponerantz's  criticism  of  the  propositional 
•ceoisit: 


The  imagery  account  seems  sooet^at  plausible 
and  relatively  straightforward.  The  propositional 
account  seems  less  satisfactory:  Aside  from  the 
problem  of  not  knowing  how  to  represent  the  letters 
in  the  first  place,  it  is  not  clear  why  rotation  is 
gradual  in  such  a system.  It  should  be  especially 
easy  to  rotate  an  image  180”  because  all  the 
relations  could  simply  be  reversed  right 
becomes  left) . To  rotate  an  image  45^  should  be 
more  difficult,  because  more  complex  substitutions 
must  be  implemented.  Nevertheless,  subjects  take 
longer  to  rotate  an  Imaged  object  180°  than  45°. 
It  appears  that  people  do  not  (or  cannot)  skip  from 
one  orientation  of  an  image  directly  to  another, 
but  must  proceed  gradually.  Such  a prediction  does 
not  follow  from  basic  concepts  of  propositional 
representation . ( p .69 ) 


But  one  can  ask  why  rotation  of  the  image  must  be  gradual.  Vftiy  should 
it  be  computationally  harder  in  the  image  model  to  calculate  a 180° 
step  than  a 1°  step?  In  terms  of  nunber  of  CPU  cycles  there  would  be 
no  differences  in  a simulation  program  like  that  of  Kosslyn  & Shwartz. 
It  is  no  less  ad  hoc  to  propose  this  limitation  on  the  image  model  than 
it  is  to  propose  it  for  the  propositional  model. 

Effects  of  Image  Size  and  Complexity 

Kosslyn  (1975)  has  found  that  sii>jects  take  longer  to  verify 


that  an  imaged  object  has  a certain  property  if  they  are  instructed  to 
m^e  the  image  of  the  object  small.  Kosslyn  argues  that  there  is  a 


"grain"  limitation  on  the  mental  image  and  that  it  is  difficult  to 
properly  represent  the  details  of  a small  image.  He  relates  this  to 
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the  supposed  perceptual  fact  that  it  is  difficult  to  discriminate 
details  of  small  objects  when  presented.  (However,  Kosslyn  has  not 
established  that  there  is  this  perceptual  phenomenon  over  the  range  of 
sizes  he  is  manipulating.)  Kosslyn  has  also  shown  that  there  is  a 
complexity  limitation  on  an  image — that  it  takes  longer  to  verify  that 
an  object  has  a certain  property  when  it  is  imaged  along  with  a more 
complex  object. 

These  results  can  be  accounted  for  by  a propositional  model 
which  assimes  that  a subject  activates  fewer  propositions  to  rep>resent 
an  object  when  instructed  to  image  it  small  and  that  he  can  activate 
fewer  propositions  when  he  must  represent  another  complex  object. 
However,  Kosslyn  and  Pomerantz  argue  that  this  propositional  account 

...seems  less  satisfactory  than  the  imagery 
account:  Why  should  people  access  less  information 
about  an  object  when  asked  to  "image  it  small." 

This  seems  ad  hoc;  a propositional  model  would  not 
lead  one  to  expect  such  effects.  (p.71) 

However,  it  is  no  less  ad  hoc  to  propose  that  small  images  suffer  a 
grain  limitation.  A priori,  one  might  have  supposed  that  smaller 
images  have  smaller  grain.  It  is  entirely  ad  hoc  to  propose  a 
complexity  limitation  on  an  image.  A property  frequently  ascribed  to 
images  (e.g.  Cooper,  1975)  is  that  they  can  be  processed  in  an 
unlimited  capacity  parallel  manner.  In  contrast,  there  does  exist  a 
propositional  model  (ACT  - Anderson,  1976)  which  incorporates  a 
complexity  limitation  on  the  nixnber  of  propositions  that  may  be  active. 
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this  limitation  was  motivated  on  grounds  entirely  independent  of 
Kosslyn's  result. 

there  is  nothing  intrinsic  to  an  image  or  to  a propositional 
representation  which  would  have  led  one  to  predict  Kosslyn's  results  of 
size  and  of  complexity.  Either  representation  requires  additional, 
non-trivlal  process  assumptions  to  predict  these  results,  there  is  no 
Justifiable  basis  for  calling  one  set  of  assunptions  more  arbitrary 
than  the  other. 

Judgments  of  Magnitude 

there  are  experiments  (e.g.  Moyer,  1973;  Paivio,  1975)  which 
show  that  when  one  is  Judging  two  mental  objects  with  respect  to  a 
dimension  on  which  they  vary,  reaction  time  decreases  with  the 
separation  of  these  objects  on  that  dimension.  In  a model  experiment, 
Paivio  (1975)  had  subjects  Judge  vAiich  of  two  items  (e.g.  a lanp  or  a 
zebra)  was  larger.  In  one  condition  Paivio  presented  the  words  and  in 
another  condition  Paivio  presented  a picture  of  the  objects  in  their 
true  size  ratio.  Reaction  time  to  make  this  Judgement  decreased  in 
either  case  as  the  items  named  or  depicted  increased  in  size  disparity. 
Paivio 's  picture  condition  provides  the  needed  control  to  support  his 
claim  that  he  is  getting  the  same  effect  with  word-cued  representations 
as  he  would  get  with  picture-cued  representations.  (It  is  somewhat 
disturbing,  lx>wever,  that  the  size  of  the  effect  is  about  twice  as 
large  with  word-cued  Judgments  as  it  is  with  picture-cued  Judgments.) 
Paivio  argues  that  this  is  evidence  that  picture  information  is 
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represented  in  an  analogue  Image  format  rather  than  a discrete 
propositional  format. 

Ihere  is  some  dispute  about  the  exact  mechanisms  underlying 
these  comparative  judgements  (Banks,  Clark,  and  Lucy,  1975;  Holyoak, 
1977).  However,  a more  relevant  point  to  the  issue  at  hand  is  that 
there  is  nothing  incompatible  with  the  notion  of  a propositional  model 
and  the  idea  that  judgement  time  can  vary  with  object  magnitude. 
Magnitudes  can  easily  be  represented  in  a propositional  model  as 
argixnents  of  propositions.  (It  is  easy  to  provide  predicate  calculus 
treatments  of  many  physical  and  mathematical  domains  that  deal  in 
continuously  varying  quantities.)  There  is  no  reason  why  the  distance 
effects  reported  could  not  come  from  a process  defined  on  these 
propositional  representations.  Of  course,  there  is  no  reason  to  have 
expected  the  effect.  However,  there  is  no  reason  to  have  expected  the 
effect  given  an  image  representation,  either.  The  reason  to  have 
expected  the  effect  comes  from  a knowledge  of  the  results  obtained  in 
perception.  However,  as  noted  earlier,  the  products  of  perception  can 
either  be  represented  in  a picture  format  or  a propositional  format. 
unlimited  Capacity  Image  Processing 

There  are  a number  of  experiments  (Cooper,  1975;  Nielson  & 
Salth,  1972;  Smith  & Nielson,  1970)  which  lead  to  the  conclusion  that 
images  can  be  matched  to  perceptual  objects  as  templates  or  in  an 
unlimited  capacity  parallel  manner.  For  instance,  Nielson  & Staith 
(1970)  find  subjects  unaffected  by  complexity  in  recognizing  faces  when 
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they  have  encoded  the  faces  as  whole  units  but  not  vAien  they  have 
encoded  the  faces  as  verbal  strings.  This  finding  has  been  used  as 
evidence  for  image  over  propositional  representations.  However,  there 
is  nothing  at  all  incompatible  in  having  propositional  representations 
and  parallel  processes.  The  ACT  model  (Anderson,  1976)  is  an  example 
Of  a propositional  system  in  which  all  basic  processes  are  parallel  and 
some  are  of  effectively  unlimited  capacity.  The  spreading  activation  of 
Quillian's  (1969)  TLC  model  can  be  conceived  of  as  an  unlimited 
capacity  parallel  process  (but  see  Collins  & Loftus,  1975). 

There  are  two  aspects  of  this  phenomenon  that  are  potentially 
troublesome,  however.  The  first  is  the  possibility  that  unlimited 
capacity  parallel  processes  can  be  obtained  with  encodings  of  visual 
Information  but  not  with  verbal  encodings.  If  one  abstract 
propositional  encoding  underlies  all  memory,  it  would  be  hard  to  see 
\t\y  different  principles  would  govern  the  representation  when  it 
encodes  information  from  different  sources. 

Second,  the  very  phenomenon  of  truly  unlimited  capacity 
parallel  processing  is  puzzling.  I know  of  no  well-understood  device 
that  could  make  recognition  discriminations  such  as  found  above  without 
an  increase  in  processing  time  after  some  bound  on  object  complexity 
was  exceeded.  There  is  sometimes  expressed  a belief  that  analogue 
processes,  because  of  their  continuous  nature,  provide  an  infinite 
amoizit  of  information.  However,  discrlminability  limitations  on  a 
system's  ability  to  respond  to  smtill  differences  imply  a finite  bound 
on  the  amount  of  information  per  analogue  process. (xxxx) 
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It  Is  possible  to  have  a computational  system  that  appears  to 
have  "unlimited  capacity"  within  a certain  range.  For  instance,  one 
can  have  a large  nunber  of  parallel  processors  only  some  of  which  are 
used  on  simple  problems.  Ihe  greater  difficulty  of  more  complex 
problems  can  be  masked  by  recruiting  the  unused  processors.  A 
complexity  effect  would  only  be  observed  for  problems  that  were 
sufficiently  complex  to  exhaust  the  available  processors. 

Template  matching  is  frequently  given  as  an  example  of 
unlimited  capacity  parallel  processing.  However,  templates  have  only  a 
certain  fineness  of  discrimination.  Faced  with  distinguishing  anong 
objects  that  require  finer  discrimination,  either  a template  matching 
program  will  make  errors  or  more  computational  resources  will  have  to 
be  allocated  to  permit  a finer  grid  to  be  computed. 

There  are  a number  of  possible  responses  to  reports  of 
inlimlted-capacity  parallel  processing  of  imsges.  One  would  be  to  give 
up  the  principle  of  finiteness  of  human  computing  resources.  A better 
response  is  to  assune  that  the  complexity  has  not  been  pushed  far 
enough — that  we  are  still  in  the  range  where  the  capacity  of  a finite 
set  of  parallel  processes  has  not  been  fully  exploited.  A third 
response  is  to  quibble  with  the  demonstrations  of  unlimited  parallel 
processing.  It  may  be  that  less  information  is  being  processed  about 
the  more  complex  objects  but  that  the  tests  fall  to  bring  out  evidence 


for  this  degradation  in  processing. 

Consider  the  experiment  by  Cooper  and  Podgorny  (1976)  on 
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recognition  of  Attneave  polygons:  they  varied  the  number  of  points  in 
the  polygons  fraa  6 to  24  and  obtained  no  effect  of  this  measure  of 
complexity  on  recognition  time.  As  distractors,  they  used 
transformations  of  the  target  polygons  with  some  of  their  points 
perturbed.  There  were  no  more  errors  made  to  distractors  for  polygons 
of  greater  complexity.  This  was  used  as  evidence  against  the 
possibility  of  a less  careful  processing  of  the  more  complex  rhapes. 
However,  the  more  complex  distractors  also  had  more  points  perturbed. 
Therefore,  one  would  have  to  remember  a smaller  portion  of  the  points 
fhom  the  more  complex  figures  to  achieve  the  same  probability  of 
detecting  a distractor. 

Cooper  & Podgorny  had  subjects  rate  similarity  of  the 
distractors  to  the  targets.  The  distractors  were,  on  the  average,  of 
equal  similarity  for  the  targets  of  different  complexity.  Apparently, 
more  points  have  to  be  perturbed  in  a more  complex  polygon  to  achieve 
the  same  difference  in  judged  similarity.  However,  to  have  shown 
uilimlted  capacity  processing  they  should  not  have  used  distractors  of 
equal  similarity  but  rather  distractor:*  that  had  the  same  nunber  of 
points  perturbed  and  by  the  same  amount.  This  would  be  a test  of 
whether  all  the  information  in  the  target  was  being  processed  in  the 
complex  figures.  To  perform  the  experiment  in  this  manner  would  mean 
that  distractors  of  greater  similarity  would  have  to  be  used  for  more 
complex  stimuli.  Cooper  and  Podgorny  found  that  there  were  more  errors 
for  more  similar  distractors.  So,  presUnably  in  the  prescribed 
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experiment  they  would  find  that  subjects  were  not  performing  as  well 
with  more  complex  figures,  arguing  against  unlimited  parallel 
processing . 

General  Analysis  of  the  Problem  of  Discriminating  Among  Representations 
I have  been  arguing  that  the  many  exemples  proposed  by  image 
theorists  do  not  provide  greater  evidence  for  an  image  rej>resentation 
than  they  do  for  a propositional  representation.  In  each  case  it  was 
foind  that  evidence  for  a particular  representation  was  really  evidence 
for  a particular  process  and  that  there  was  no  good  reason  to  associate 
the  process  with  a particular  representation.  (There  were  important 
second  order  complications  with  the  unlimited  capacity  issue.)  Similar 
difficulties  were  also  shown  with  the  arguneits  given  by  propositional 
theorists.  I would  like  to  formalize  the  general  point  that  these 
examples  illustrate.  This  general  point  is  that  it  is  not  possible  for 
behavioral  data  to  decide  uniquely  issues  of  internal  representation. 
The  reason  is  that  one  cannot  Just  test  questions  about  a 
representation  in  the  abstract.  One  must  perform  tests  of  the 
representation  in  combination  with  certain  assunptions  about  the 
processes  that  use  the  representation.  That  is,  one  must  test  a 
representation-process  pair.  One  can  show  that,  given  a set  of 
assumptions  about  an  image  representation  and  a set  of  processes  that 
operate  on  it,  one  can  construct  an  equivalent  set  of  assunptions  about 
a propositional  representation  and  its  processes.  Or  one  can  be  given 
a propositional  theory  and  construct  an  equivalent  imagery  theory.  In 
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fact,  it  is  possible  to  establish  a more  general  claim:  Given  any 
representation-process  pair  it  is  possible  to  construct  other  pairs 
with  different  representations  whose  behavior  is  equival«it  to  it. 
These  pairs  make  up  for  differences  in  representation  by  assuming 
compensating  differences  in  the  processes. 

The  argument  that  I will  give  to  establish  this  claim  has  been 
changed  somewhat  from  an  argunent  to  the  same  point  given  in  Anderson 
(1976).  It  has  been  changed  both  to  make  the  claim  more  general  and  to 
try  to  make  it  clearer  why  it  is  possible  to  achieve  equivalences  among 
different  representations. 

Note  to  typesetter:  [I]  should  be  set  in  script. 

Suppose  one  had  a theory  of  internal  representation,  Cll.  This 
theory  would  specify  a set  (probably  infinite)  of  possible  internal 
repi'esentations  which  we  can  denote  I^,l2,...,Ijj....  However,  a theory 
which  only  specifies  internal  representations  says  virtually^  nothing 
about  behavior  and  therefore  is  not  testable  by  itself.  One  must 
specify  some  processes  that  operate  on  these  internal  representations 
in  order  for  behavior  to  occur.  It  is  useful  to  discriminate  among 
three  types  of  processes.  There  are  encoding  processes  which  map 
external  stimuli,  S^,  into  internal  representations,  Ij.  These 
encoding  processes  will  be  denoted  by  the  function  E.  The  operation  of 
encoding  a stimulus  can  be  represented  E(Sj)sIj.  Second,  there  are 
processes  of  internal  transformation  which  can  be  represented  by  the 
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function  T.  The  notation  T(Ij^)=I.  represents  a transformation  from 
internal  representation  to  Ij.  Finally,  there  are  decoding 
processes  specifying  how  internal  representations  are  manifested  as 
responses.  So  if  state  results  in  response  Rj,  this  will  be  denoted 
RjsD(I^). 

This  is  a quite  general  framework  for  representing  cognitive 

theories  and  I do  not  think  it  blurs  any  significant  issue  involved  in 

the  image-propositional  controversy.  Figure  3a  represents  a possible 

scheme  of  information  processing  within  this  framework.  We  have  three 

stimuli  Sj,  and  being  presented,  not  necessarily  at  the  same 

time.  By  the  encoding  process  E,  these  result  in  internal 

representations  I^,  Ij,  and  I^.  and  Ij  result  by  T in  an 

intermediate  internal  representation  I^.  To  preserve  the  notation  of  T 

mapping  one  representation  into  another,  we  will  assvine  that  a 

conjunction  of  internal  representations  is  itself  an  internal 

representation.  So  T(Ij^4Ij)=I|jj.  There  are  other  transformations 

illustrated  which  occur  at  various  points  of  time:  T(I.)=I  , 

J n’ 

T(I^4Ij^)sIp.  Finally,  the  decoding  process  maps  and  into 

response  R. 

Insert  Figure  3 about  here 

If  we  restrict  ourselves  to  behavioral  data,  we  cannot  directly 
observe  the  internal  processes,  E,  T,  and  D nor  the  internal 
representations.  All  we  observe  is  that  at  various  times  the  stimuli 
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Sj,  Sj  and  arrive  and  that  sometime  later  response  R is  emitted. 
The  question  of  interest  is  whether  behavioral  data  (i.e.,  observation 
of  the  contingencies  between  such  events  and  the  time  of  these  events) 
are  adequate  to  constrain  a theory  of  internal  representation.  Such  a 
theory  of  representation  will  be  part  of  a model,  M,  that  also 
specifies  the  processes  that  operate  on  the  representation.  It  will  be 
argued  that  models  with  very  different  theories  of  representation  can 
perfectly  mimic  the  behavioral  predictions  of  M.  These  alternative 
models  will  compensate  for  differences  in  the  representation  by 
different  assumptions  about  the  processes.  Therefore,  these  models  are 
not  discriminable  from  M on  the  basis  of  behavioral  data.  Therefore, 
the  representation  assumed  by  M is  not  discriminable  from  the  very 
different  representations  assumed  by  the  other  models. 

Let  Cl]  be  a specification  (e.g.,  a grammar)  of  the  possible 
representations  I^,l2i... under  M.  [I]  by  itself  is  not  a very 
interesting  theory  of  representation.  For  instance,  to  assert  that  our 
representation  was  predicate  calculus  would  not  be  very  informative 
unless  we  had  some  idea  what  events  would  lead  to  a representation  like 
giveCJohn,  Mary,  ball).  It  would  be  a very  different  theory  of 
representation  if  this  formula  were  used  to  represent  the  meaning  of 
"Fred  eats  pizza"  than  if  it  were  used  to  represent  "John  gives  the 
ball  to  Mary."  That  is,  we  want  a theory  of  representation  to  specify 
how  various  inputs  are  represented.  A theory  of  representation  should 
include  the  encoding  process,  E.  This  forces  a connection  between  our 
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I representation  and  the  external  v#orld.  Let  us  refer  to  the  pair 

I <[!],£>  as  M’s  theory  of  representation.  The  pair  <T,D>  will  be  M's 

theory  of  the  processes  that  operate  on  the  representation.  Vihat  I 
will  show  is  that  given  a different  theory  of  representation  <[!]*,£*>, 

one  can  embed  that  theory  in  a different  model  M*  with  different 

« « • 

processes  <T  ,D  > and  have  M mimic  M in  all  its  predictions  about 
behavioral  data. 

Preservation  of  Internal  Distinctions 

It  is  not  the  case,  however,  that  any  theory  of  representation 
can  be  made  part  of  a model  M that  mimics  M.  The  theory  of 
representation  must  satisfy  a condition  called  preservation  of  internal 
distinctions;  ttie  theory  of  representation  <[!]*,£*>  preserves  the 
Internal  distinctions  of  another  theory  <[!],£>  if  (a)  there  is  a one- 
to<<xie  mapping  f from  [I]  to  [I]  such  that:  (b)  f has  a computable 
inverse  which  will  be  called  f“\  (c)  f”^  will  map  the  encoding  of  S in 
the  mimicking  theory,  £ (S),  into  the  encoding  of  S in  the  target 
theory,  £(S),  i.e.,  £(S)sf”^(£  (S))  for  all  S.  The  fact  that  the 
tapping  is  one-to-one  assures  that  any  distinctions  among 
representations  in  [I]  will  be  preserved  in  distinctions  among 
representations  in  [I]*.  The  fact  that  £(S)sf’^(E*(S))  assures  that  £ 
and  E assign  corresponding  representations  to  the  same  stimuli.  One 
consequence  of  these  requirements  is  that  for  and  Sj  if 
E (S^)sE  (Sj)  then  £(Sj^)s£(Sj) . That  is,  the  encoding  process  £*  does 
not  fail  to  preserve  any  of  the  distinctions  among  stimuli  that  are 
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preserved  by  the  process  E.  It  will  prove  to  be  necessary  that  f have 

• • 

a CGuputable  inverse  to  guarantee  that  process  assiinptions  <T  ,D  > of 
the  mimicking  model  M can  be  computed.  One  would  not  want  to  propose 
processes  which  could  not  be  computed  and  hence  (by  Turing's  thesis  or 
Church's  thesis — see  Minsky,  1967)  were  not  capable  of  specification. 

While  the  condition  of  preservation  of  internal  distinctions 
excludes  some  representations  from  the  guarantee  of  mimicry,  it  is  not 
so  severe  as  to  exclude  all  interesting  possibilities  from  mimicry.  In 
particular,  we  could  have  a propositional  model  mimic  a picture  model. 
The  mapping  f in  this  case  would  transform  Uie  picture  into  a complete 
propositional  description.  (I  assune  it  has  been  generally  conceded 
that  all  the  information  in  a picture  can  be  propositional Ized.)  It  is 
also  the  case  that  a propositional  model  can  be  mimicked  by  a dual  code 
model  like  Figure  2.  In  this  case,  f would  map  a propositional 
representation  into  a dual  code  representation  that  contained  the  same 
information . 

One  might  wonder  under  what  conditions  the  mapping  would  have  a 

computable  inverse.  If  E and  E*  are  primitive  recursive  (Minsky, 

1967) f a mapping  f can  be  constructed  with  a computable  inverse  f~\ 

• -1  • -1 

One  can  simply  make  f:E  •E  where  E*E  denotes  the  combination  of 

applying  first  the  inverse  of  E and  then  E*.  Similarly,  f”^sE»E*”\ 

• -1  *-1 
If  E and  E are  primitive  recursive  E and  E will  be  computable  and 

hence  f and  f~^  will  be  computable. Note  that  the  above  argunent 


I 


establishes  that  f and  its  inverse  are  computable  by  showing  one  way  to 


1 
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compute  f,  l.e.,  E*eE‘’\  It  does  not  mean  that  this  is  the  best  or 
only  way  to  compute  f.  For  instance,  in  mapping  fVom  a picture 
representation  to  a propositional  representation  there  would  be  no  need 
to  map  to  the  external  stimulus  by  E~^  and  then  to  the  propositional 
representation  by  E . A more  direct  propositional  encoding  of  the 
picture  should  be  possible. 

It  should  also  be  noted  that  for  any  theory  of  representation 
<[I],E>  liiere  E has  a computable  inverse,  there  is  at  least  one  other 
representational  theory  that  will  mimic  <[I],E>.  This  is  the  theory 
that  assigns  to  each  stimulus  a representation  isomorphic  to  that 
stimulus.  For  this  theor> , the  mapping  fsE"^  satisfies  the  condition 
of  preservation  of  internal  distinctions.  Since  E need  only  be 
restricted  bo  primitive  recursive  to  have  a computable  inverse,  this 
means  virtually  any  theory,  not  just  imagery  and  propositional,  can  be 
mimicked . 

A final  feature  to  note  is  that  if  the  map  f ftrora  [1]  to  [I]* 
is  a function  (i.e.  its  inverse  is  1-1)  then  there  is  a potential  for 
mutual  mimicry.  That  is,  theories  using  <[I]*,E*>  can  be  mimicked  by 
theories  using  <tI],E>  as  well  as  the  converse.  I would  argue  that 
this  mutual  mimicry  holds  between  dual  code  theory  and  propositional 
theory  because  it  seems  that  one  can  define  a 1-1  function  from  a dual 
code  representation  to  a propositionalization  of  it. 

Proof  of  Behavioral  Mimicry 

It  remains  to  be  established  that  there  exists  the  potential 
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for  mimicry  given  a 1-1  map  f.  I will  show  something  stronger — not 
only  is  it  possible  to  mimic  the  observable  behavior  of  M using  <CI3,E> 
with  a model  M using  <[I]  ,£>*,  but  it  is  possible  to  produce  a set  of 
Internal  operations  on  the  representations  in  [1]*  isomorphic  to  the 
Internal  operations  on  [I].  Figure  3b  illustrates  this.  This  is 
important.  Since  the  mimicking  model  goes  through  the  exact  sane 
steps,  not  only  will  it  reproduce  the  same  behavior,  it  will  reproduce 
the  same  time  relationships.  That  is,  it  will  mimic  such  things  as 
reaction  times. 

After  all  these  preliminaries,  it  only  takes  a few  brief 

• • • 

remarks  to  establish  that  E , T and  D will  mimic  E,  T,  and  D Just  as 
Figire  3 illustrates.  If  the  encoding  operation  E maps  the  stimulus  S 

into  I then  E maps  S into  f(I).  By  its  construction,  the  map  f was 

* 

guaranteed  to  convey  on  E this  mimicry  of  E.  It  is  also  possible  to 
construct  the  elements  T*  and  D*  to  achieve  mimicry  of  T and  D.  That 

is,  if  T maps  1^^  into  1^,  T*  will  map  f(Ijj)  into  f(Iy)  and  if  D maps  I 

s • 

into  R,  D will  map  f(I)  into  R.  One  T to  achieve  this  can  be 

constructed  as  f»T*f”^  where  this  denotes  the  operation  of  f"^  on  the 

internal  representation,  then  the  operation  of  T,  then  the  operation  of 

f.  Since  f,  f”^  and  T are  computable,  so  is  T*.  If  T(I  )sl  , then 

* * 

T*(f(Ijj))sf*T*r^(f(Ijj))sr#T(Ijj)=f(Iy).  That  is,  if  T maps  1^^  into  1^, 
T will  map  f(Ijj)  into  f(Iy).  Similarly,  we  can  designate  D*  as  D»f“\ 
D*  will  also  be  computable. 

Oie  might  object  to  the  complicated  specification  of  T*  as 
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r*T»r  and  of  D as  D»r  . However,  it  does  not  follow  that  these  are 

• • 

the  simplest  specifications  of  T or  D . This  demonstration  has  the 

character  of  an  existence  proof.  I have  shown  that  there  exists  at 

• • 

least  one  process  pair  <T  ,0  > which  will  combine  with  the 

« • 

representation  <[IJ  ,E  > to  yield  mimicry  of  M.  For  all  we  know,  there 
is  a <T  ,D  > which  is  much  simpler  than  <T,D>  fhom  M.  This  argunent 
establishes  nothing,  one  way  or  the  other,  about  the  relative  parsimony 
of  the  two  simplest  models  for  representations  [I]  and  [I]*.  The  point 
of  this  section  is  to  establish  that  there  are  not  purely  behavioral 
criteria  for  distinguishing  between  the  two  theories.  Certainly,  if  it 
could  be  shown  for  some  T that  its  simplest  T*  mimic  was  f»T»f“\  there 
would  be  parsimony  grounds  for  rejecting  it.  However,  in  actual 

examples  such  as  the  forthcoming  illustration,  there  has  always  turned 
out  to  be  simpler  T ^)ecifications.  A more  thorough  evaluation  of 
parsimony,  an  unfortmately  subjective  concept,  comes  later  in  the 
paper. 

A similar  remark  can  be  made  about  time  relationships.  To 

• a 

guarantee  M can  mimic  H we  must  assime  the  time  for  each  step  of  T in 

Figire  3 is  identical  to  the  time  for  the  corresponding  step  of  T.  For 
this  to  be  so  and  since  we  are  only  assured  of  one  specification  of  T* 
as  f^Taf^  we  have  to  assime  that  the  time  for  M*  to  perform  the 
sequence  f*T*f“^  can  be  made  identical  to  the  time  for  M to  perform  T. 
This  can  be  done  by  assiming  that  M*  can  compute  T faster  than  M and 
also  can  rapidly  compute  f and  f“\  Thus,  by  speeding  up  the  operation 
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of  M we  can  always  place  Its  operations  in  time  step  with  M.  We  might 
be  prevented  from  such  speed-up  proposals  if  we  had  adequate  knowledge 
about  possible  physiological  implementation  but  we  do  not.  (An 
evaluation  of  the  potential  of  physiological  criteria  also  follows.) 
Moreover,  as  it  seems  we  can  usually  construct  T*  more  simply  than  its 
form^ly  guaranteed  specification,  the  need  for  such  speed-up  proposals 
does  not  seem  to  arise  in  practice. 

In  concluding  this  argtnent,  I refer  the  reader  back  to  Figure 
3a  aond  3b  which  capture  its  essence.  That  shows  how  there  can  be  an 
isomorphism  between  the  operations  on  two  distinct  representations  in  M 
and  M . Some  people  have  the  feeling  that  if  the  operations  are 
isomorphic  the  distinction  between  M and  M*  becomes  almost  meaningless. 
I would  like  to  endorse  this  attitude.  It  embodies  an  important  claim 
of  this  paper:  At  a useful  level  of  abstraction  there  need  be  no 
difference  in  the  behavior  of  systems  that  use  a wide  range  of 
representations.  In  particular,  there  need  be  no  difference  in  the 
behavior  of  a system  which  uses  a propositional  versus  a pictorial 

Q 

representation . ^ 

Another  counterargument  to  this  argument  about  representational 
indeterminacy  is  that  it  misses  an  impnrtant  distinction.  While  it  may 
be  true  that  a propnsitional  representation  can  be  modified  to 
accomodate  data  predicted  by  a dual-code  theory  (or  vice  versa) , the 
couiterargument  goes,  it  is  still  the  case  that  one  theory  predicted 
the  result  and  the  other  theory  was  modified.  There  are  some  points 


k 
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that  need  to  be  made  here.  First,  in  practice,  it  is  often  not  the 
case  that  one  representation  rigorously  predicted  the  result.  Rather 
it  is  the  case  that  we  "intuit”  the  result  as  being  implied  by  the 
representation.  Second,  in  those  uses  of  fairly  rigorous  predictions, 
the  derivation  depends  critically  on  process  assumptions.  Therefore, 
it  is  not  evidence  for  the  representation  per  se,  but  for  a 
representation-process  pair.  It  is  correct  to  take  this  predictive 
power  as  evidence  for  that  representation-process  pair.  However,  it 
certainly  seems  incorrect  to  take  the  outcome  as  evidence  for  the 
representation  in  abstract  since  conjoined  with  other  process 
assunptions  the  representation  would  lead  to  very  different 
predictions.  Finally,  it  is  wrong  to  conclude  ft*om  a theory's  failure 
to  predict  a result  that  the  theory-builder  should  do  anything  more 
than  appropriately  modify  his  theory.  Any  theory  proposed  today  must 
be  wrong  in  many  aspects.  Any  worthwhile  theory  is  based  on  and 
accounts  for  a good  range  of  data.  Therefore,  the  appropriate  response 
to  inevitable  discrepancy  usually  should  be  modification  of  the  theory 
not  abandonment.  The  appropriate  reason  for  abandoning  a theory  rather 
than  modifying  it  is  when  an  alternate  theory  is  developed  that  much 
more  parsimonously  accounts  for  the  data.  The  parsimony  disadvantage 
of  a theory  may  arise  fnom  modifications  to  account  for  discrepant 
results.  However,  such  modifications  also  may  not  leave  the  theory  at 
any  parsimony  disadvantage. 

A natural  reaction  to  this  argument  is  to  view  it  as  "so  much 
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blind  formalian."  But  considered  in  the  light  of  the  prior  discussion 
About  attempts  to  discriminate  between  propositional  and  imagery 
representation,  the  argument  seems  more  compelling.  That  discussion 
contained  frequent  evidence  for  the  central  claim  of  this  formal 
argument:  Differences  in  representation  can  be  compensated  by 
differences  in  the  processes  that  operate  on  these  representations.  I 
would  also  like  to  apply  this  formal  argunent  to  a specific  case: 
Example  of  Equivalence 

This  argunent  has  been  made  in  quite  abstract  terms.  This 
abstractness  was  required  to  be  able  to  establish  the  generality  of  the 
point  about  non-identifiability.  However,  it  is  hard  to  recognize  the 
force  of  the  argunent  because  of  its  abstractness  and  also  because  of 
the  Importation  of  the  terminology  of  primitive  recursive  functions  and 
the  need  to  create  such  formal  constructions  such  as  T*=f»T*f”\ 
Therefore,  I will  go  through  an  exanple  of  i'jov  this  result  can  be 
mapped  into  the  reality  of  a contemporary  psychology  experiment. 

The  task  I have  chosen  is  rotation  of  letters  as  studied  by 
Cooper  and  Shepard  (1973)  • I will  sketch  out  an  imagery  model 
(representation  plus  process)  of  this  phenemena,  a propositional 
representation,  and  use  the  above  analysis  to  construct  a process 
i4iich,  operating  on  the  propositional  representation  will  mimic  the 
behavior  of  the  imagery  model.  I will  assune  that  the  imagery 
representation  [I]  of  a letter  is  a two-dimensional  matrix  encoding  in 
terms  of  1's  and  O's  whether  particular  squares  are  filled  in.  This  is 


Mental  Imagery  <17 


basically  the  stimulus  display  presented.  Therefore,  the  encoding 
process  for  an  imagery  model  is  one  of  identity— i.e.,  E=  I.  We 
assumed  stored  in  memory  is  an  upright  representation  of  the  letter.  A 
test  object  is  presented  and  encoded  into  its  matrix  form.  In  the 
Cooper  4 Shepard  paradigm  the  presented  object  is  usually  not  in 
upright  form.  The  subject  must  decide  whether  the  presented  object 
matches  a letter  or  is  a mirror  image  of  a letter  (e.g.  R vs.^  ).  We 
assume  that  the  matrix  representation  of  the  presented  object  is 
rotated  in  til  it  is  in  upright  form.  This  corresponds  to  T,  the 
process  of  internal  transformation.  It  is  easier  to  tighten  our 
application  of  the  formalism  if  we  assixne  the  rotation  proceeds  in 
discrete  steps  (as  is  done  by  Kosslyn  4 Shwartz  in  their  simulation). 
So,  in  each  discrete  time  unit,  matrix  is  transformed  into  matrix  Mg 
where  these  two  are  related  by  a small  angle  of  rotation  around  an  axis 
through  the  center  of  the  matrix.  We  can  denote  this  as  MgsTCH^).  It 
should  be  obvious  how  to  spell  out  the  operation  T (e.g.,  as  a computer 
program).  The  decoding  operation  applies  when  the  rotated  matrix  are 
in  upright  orientation.  There  is  a technical  difficulty  in  deciding 
when  the  matrix  is  upright,  but  I assune  as  others  have  that  there  are 
features  which  will  allow  one  to  decide  that  a character  is  upright 
before  one  has  recognized  it.  The  recognition  operation  consists  of 
computing  a complete  match  between  the  rotated  matrix  and  the  stored 
matrix.  This  is  measured  by  the  nunber  of  0-1  correspondences  in  the 
two  matrices.  If  the  stimuli  offer  a close  enough  match,  a positive 
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response  is  emitted  and,  if  not,  a negative  response  is  emitted.  Let 
0(M^,M2)  be  some  scaling  of  the  distance  between  the  filled  cells  of  M^ 
and  the  closest  filled  cells  of  M2.  Then  we  have  D defined  with 
respect  to  the  prototype  letter  matrix  Mp: 

D(M)  = YES  iff  0(M,Mp)  > C 
= NO  iff  0(M,Mp)  < C. 

The  above  may  or  may  not  correspond  to  one's  favored  interpretation  of 
an  imagery  model  for  this  task.  For  present  purposes  of  illustration 
it  is  irrelevant  whether  it  does. 

Insert  Figure  H and  Table  1 about  here 

Let  us  now  develop  a propositional  representation  for  letters. 
Figure  4 represents  the  letter  R in  standard  orientation  and  Table  1 
gives  its  propositional  code  representation.  The  main  proposition, 

(R,  SI,  S2,  S3,  S4,  S5,  S6)  defines  R in  terms  of  its  component 
line  sej^ments.  Other  propositions  define  these  line  segments  according 
to  their  shape  and  terminal  points.  Others  give  the  length  of  the 
segments.  Others  give  the  angles  between  straight  lines.  Of 
particular  interest  is  the  proposition  AXIS  (R,  P3)  which  identifies  P3 
as  giving  the  axis  of  orientation  for  the  letter  R and  the  VECTOR-ANGLE 
propositions  which  identify  the  locus  of  all  the  crucial  points 
relative  to  this  axis  propositional  code.  To  complete  our 
propositional  representation  we  need  to  specify  E*  the  encoding 
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process.  This  Is  the  operation  by  which  one  goes  from  the  external 
stimulus  (e.g.,  a matrix)  to  a propositional  encoding  in  terms  of 
lines.  This  process  is  basically  one  of  picture  parsing  for  which 
there  do  exist  available  algorithms  (e.g.,  Duda  & Hart,  1973). 
Assuming  clean  displays  and  a small  repertoire  of  line  types  this  would 
be  a simple  algorithm.  The  algorithm  would  become  more  difficult  as 
the  displays  deviated  from  these  assumptions.  (One  remark  about  this 
picture  parser:  It  could  not  recognize  the  stimulus  configurations 
before  parsed.  Therefore,  if  Figure  4 were  an  encoding  of  a stimulus 
array,  the  R in  the  propositions  would  be  replaced  by  an  arbitrary  term 
or  node.)  The  above  may  or  may  not  correspond  to  one’s  favorite 
propositional  model  for  letter  representation.  As  with  the  imagery 
model  this  does  not  matter  for  purposes  of  illustration.  No 
endorsement  should  be  inferred  for  either  model. 

Now  we  have  to  proceed  with  the  construction  of  the  mimicking 

• • 

processes  T and  D for  the  propositional  model.  The  first  thing  to 
note  is  that  we  can  create  the  requisite  map  f between  [I]  and  [I]*  as 
E**E“\  Since  E was  the  identity  transformation  we  have  fsE*.  Its 
inverse  f“^  will  be  E*“^  or  a transformation  that  goes  fhom  a 
propositional  encoding  like  that  in  Figure  4 to  a matrix  encoding. 
Such  a function  is  clearly  computible.  The  other  condition  required 
for  preservation  of  internal  distinctions  is  that  E(S)sf"^(E*(S)) . 
Since  E is  the  Identity  and  is  E this  can  be  rewritten  as  (SsE*”^ 


lAtich  is  clearly  true. 
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Mow  let  us  consider  construction  of  T*  the  process  of  internal 
transformation  of  the  representation.  We  have  from  the  formal  results 
that  T sf*T«f  =£•!•£  . T , as  a specification  of  input>output , 
might  seem  quite  complicated.  It  might  seem  to  require  applying  £*“^ 
mapping  the  propositional  representation  to  its  matrix  form,  applying  T 
doing  the  matrix  rotation,  and  then  applying  £*  to  map  from  the  matrix 
back  to  the  line  drawing.  But  consider  the  final  effect  of  this — there 
will  be  a change  only  in  the  angle  in  the  V£CT0R-ANGL£  propositions. 
This  change  in  angle  of  orientation  could  be  directly  computed.  This 
example  illustrates  an  important  fact  of  the  formal  analyses.  One 
should  not  reason  from  a complex  sp^icification  guaranteeing  the 
existence  of  an  input-output  relation  to  the  conclusion  that  the 
computation  of  this  relation  is  complex.  So,  in  conclusion,  T*  will 
compute  small  changes  in  the  angle  of  orientation  in  step  with  the 
small  rotations  performed  by  T.  To  remind  the  reader  of  an  earlier 
point  (p.  xxx):  It  is  no  more  arbitrary  to  assune  small  slow  changes  in 
an  orientation  parameter  than  to  assune  small  slow  changes  of  rotation. 

Now  let  us  consider  the  construction  of  D*=D*f“^=D*£*”\  This 
could  be  interpreted  as  a process  which  transformed  the  propositional 
encoding  into  a matrix  format  and  applied  the  decision  function  D. 
However,  D could  be  defined  directly  with  respect  to  the  propositional 
encoding.  Some  propositional  matching  algorithm  (e.g.,  Anderson,  1976) 
could  be  used  to  obtain  the  degree  of  structural  match  between  the 
probe  aid  the  prototype.  There  would  also  have  to  be  a scaling  of  the 
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match  between  the  lengths  and  angles  in  the  prototype  and  the  probe. 
The  only  difference  between  a letter  and  its  mirror  image,  once  rotated 
into  congruence  would  be  the  value  of  the  vector-angles.  Thus,  we  can 
define  a function  0*  mapping  the  disparity  between  Rp,  the 

prototype  representation,  and  R,  a particular  representation.  We  can 

a 

define  D as: 

D*(R)  = yes  iff  0*  (Rp  R)_>  C* 

s no  iff  0*  (Rp,R)  < C*. 

For  this  application  0 need  not  be  constructed  to  produce  an 
identical  metric  on  differences  between  stimuli  as  0 produces.  The 

prototypes  and  their  mirror  images  will  be  quite  different  by  either  0* 

» 

and  0 and  hence  D and  D will  agree  as  to  which  are  same  or  different. 
If  the  task  were  to  discriminate  among  very  similar  stimuli,  care  would 
have  to  be  taken  to  get  0 and  0*  correspond  in  the  metric  they  imposed. 
That  they  can  be  made  to  so  correspond  is  a consequence  of  the  general 
result  on  equivalence. 

This  example  illustrates  how  one  can  generate  a propositional 
model  to  mimic  an  imaginal  model.  An  example  of  the  reverse  could  also 
be  given.  I hope  the  reader  will  agree  that  the  propositional  model 
produced  by  this  algorithm  is  plausible  relative  to  the  plausibility  of 
the  imagery  model  from  which  it  was  derived.  By  my  own  subjective 
estimate,  it  seems  more  plausible. 

Possible  Responses  to  Non-Identifiability 

There  are  at  least  two  justifiable  responses  to  this  result 
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about  non-identifiability.  One  is  to  simply  try  to  develop  some  model, 
imagery  or  propositional,  that  is  compatible  with  the  data  and  not  to 
worry  about  distinguishing  it  from  all  possible  models  with  other 
representations.  The  fact  that  there  is  no  explicit  model,  imagery  or 
propositional,  that  even  comes  close  to  accounting  rigorously  for  the 
available  data  should  indicate  that  this  is  a substantial  goal  in 
itself.  The  second  approach  is  to  look  for  other  criteria  besides 
bdiavioral  data  for  selecting  ainong  representations.  Personally,  I 
find  myself  leaning  toward  the  first  response.  I would  like  to  state  a 
defense  of  the  first  response  and  then  'iurn  to  a detailed  discussion  of 
the  second. 

There  are  a number  of  ways  to  proceed  in  research.  One  is  to 

proceed  with  a strongly  empirical  bent,  with  little  guidance  from 

work 

general  theoretical  framed  The  hope  of  this  approach  is  that  a theory 
will  somehow  emerge  from  the  empirical  work.  Sometimes  results  on  non- 
identifiability  are  used  to  defend  this  empirical  orientation  over  a 
more  theoretical  orientation.  They  should  not  be  so  used.  There  are 
two  types  of  theoretical  approaches  and  non-identifiability  results 
only  argue  against  one. 

The  theoretical  approach  damaged  by  non-identifiability  results 
is  that  vAiich  attempts  to  discover  the  "true"  theory.  This  cannot  be 
done  if  we  cannot  decide  on  issues  like  propositional,  vs.  imaginal  or 
serial  versus  parallel  (Townsend,  19711).  However,  one  can  proceed  in  a 
less  ambitious  theoretical  direction.  This  is  to  formulate  a more-or- 
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less  complete  model  without  a coomltment  to  discriminate  it  from  all 
other  possible  models.  Non-identifiability  results  imply  that  there 
are  other  models  which  will  generate  the  same  predictions.  However, 
the  important  fact  is  that  many  models  will  not  generate  the  same 
predictions.  One's  non-unique  model  is  perfectly  capable  of  being 
tested  and  proven  wrong.  In  proving  it  wrong  one  would  also  be  proving 
wrong  all  of  its  equivalent  models.  It  is  no  mean  feat  to  come  up  with 
a model  capable  of  accounting  for  the  existing  range  of  empirical 
facts.  I think  it  is  a fair  statement  that  no  current  model  handles 
the  existing  range  of  results  on  imagery.  Producing  such  a model  seems 
a more  worthwhile  endeavor  than  deciding  among  the  grand  contrasts  such 
as  imagery  versus  propositional. 

Other  Criteria  for  Identiflabil ity  Besides  Behavioral 
The  argunent  to  this  point  is  that  behavioral  data  do  not 
provide  a basis  for  deciding  between  Imagery  and  propositional 
theories.  The  range  of  possible  theories  is  so  great  that  for  any 
imagery  theory  there  will  be  a mimicking  propositional  theory  and  vice 
versa.  Past  work  that  has  claimed  to  decide  between  propositional  and 
imagery  theory  resV  on  implicit  "ground  rules"  about  what  were  the 
acceptable  variations  in  these  theories.  However,  to  date  there  are  no 
compelling  reasons  to  accept  these  ground  rules.  I have  argued  that  by 
violating  these  rules  in  quite  acceptable  ways  one  could  get  one  type 
of  structure  to  account  for  data  claimed  to  be  uniquely  explainable  by 
another  type  of  structure.  Indeed,  the  representational  issue  has 
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proven  so  slippery  because  the  ground  rules  change  over  time  and  fyoro 
researcher  to  researcher . If  it  were  possible  to  justify  some 
constraints  on  the  range  of  possible  theories  then  behavioral  data 
might  be  able  to  achieve  unique  identifiabillty.  This  last  section 
will  consider  these  types  of  non-behavioral  criteria  for  constraining 
theories.  These  are  physiological  criteria,  the  criteria  of  parsimony 
and  plausibility,  and  the  criteria  cf  efficiency  and  optimality. 
Physiological  Criteria 

An  obvious  additional  constraint  comes  from  physiological  data. 
If  we  could  open  the  brain  and  observe  it  operating  on  pictures  or  on 
propositions  it  seems  that  the  issue  would  be  settled.  There  are  tvo 
problems  with  this  solution,  one  serious  and  one  not,  but  both  worth 
considering.  The  non-serious  objection  is  to  argue  that  observations 
of  brain  functions  have  the  same  problem  of  interpretation  that 
behavioral  data  do.  Suppose,  not  just  to  be  bizarre,  we  observed  an  m 
X n grid  of  data  encoded  on  the  brain's  surface  and  that  this 
corresponded  to  a picture  of  an  object.  This  observation  is  a datvjn 
that  a theory  must  account  for.  It  would  be  possible  to  attribute  that 
datum  to  some  source  other  than  the  fact  that  such  a grid  was  actually 
neurally  encoded.  That  is,  reports  of  neural  observation  can  be 
doubted  just  as  introspective  reports  are  doubted.  However,  such  a 
level  of  skepticism  is  clearly  unacceptable,  whereas  the  skepticism 
about  the  introspective  reports  is  acceptable.  It  is  of  interest  to 
note  that  direct  observation  always  has  had  a privileged  status  as  a 
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means  of  determining  the  state  of  an  object,  even  though  it  must  be 
handled  with  some  caution  (e.g.  the  stick  that  appears  crooked  in  the 
water).  Unless  there  is  good  reason  for  believing  otherwise,  the 
working  assunption  of  all  men,  including  scientists,  is  that  there  is  a 
1-1  correspondence  between  the  structure  of  a visual  percept  and  the 
structure  of  the  object  perceived.  The  problem  with  behavioral  data 
such  as  introspection  is  that  it  is  not  direct  observation  of  the 
objects  under  consideration. 

The  serious  reason  for  challenging  physiological  data  is  that 
it  does  not  provide  anything  like  direct  observation  of  the  mental 
objects.  Use  of  it  tends  to  require  more  perilous  chains  of  inferences 
than  use  of  behavioral  data.  No  neuroscientist  has  found  anything  like 
direct  evidence  for  propositions  or  images.  This  is  not  to  say  that 
such  data  are  impossible  to  achieve  in  principle,  only  that  it  is 
unlikely.  This  is  also  not  to  say  that  the  more  indirect  data  do  not 
provide  constraints  on  a psychological  theory.  (See  Fodor,  1975,  for  a 
discussion  of  the  difficulty  in  reducing  the  concepts  of  cognitive 
psychology  to  physiological  concepts.;  However,  such  indirect  data 
cannot  decide  between  image  or  proposition. 

The  principle  physiological  data  cited  for  an  imagery  theory 
concern  hemispheric  asyrnnetry  in  Information  processing.  It  is  claimed 
that  the  right  hemisphere  is  better  adapted  for  spatial  tasks  and  that 
the  left  hemisphere  is  better  adapted  for  linguistic  and  analytic 
tasks.  Part  of  the  evidence  for  this  hemispheric  differentiation  comes 
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ft*cxD  experiments  showing  selective  loss  of  abilities  with  damage  to  one 
of  the  hemispheres.  (Blakemore,  Iverson,  & Zangwill,  1972;  Kimura, 
1963;  Meier  & French,  1965).  Evidence  for  specificity  of  function  is 
also  found  in  split-brain  patients  (Gazzaniga  & Sperry,  1967).  Also 
behavioral  experiments  (e.g.  Gif fen,  Bradshaw,  & Nettleton,  1972; 
Klatzky  & Atkinson,  1971)  have  shown  differences  in  the  speed  with 
i4)ich  tasks  can  be  performed  which  are  presented  to  the  right  visual 
field  (and  hence  directly  to  the  left  hemisphere)  versus  those 
presented  to  the  left  visual  field  (and  hence  directly  to  the  right 
hemisphere) . Tasks  that  involve  an  important  verbal  component  are 
performed  better  when  presented  to  the  right  visual  field  while  visual 
tasks  do  better  in  the  left  visual  field. 

These  studies  on  hemispheric  specialization  really  provide  very 
little  evidence  on  the  form  of  information  representation.  One  could 
propose  that  all  information  has  a propositional  form  but  that 
propositions  encoding  visual  information  are  stored  in  the  right 
hemisphere  and  propositions  encoding  verbal  in  the  left.  Another 
possibility  suggested  by  Anderson  i Bower  (1973)  was  that  rather  than 
having  the  data  differentially  stored  one  could  have  procedures 
differentially  stored.  That  is,  procedures  for  performing  verbal  tasks 
would  be  in  the  left  hemisphere  and  procedures  for  spatial  tasks,  right 
hemisphere.  Both  types  of  procedures  could  take  propositional 
information  as  their  data. 

A recent  experiment  by  Patterson  & Bradshaw  (1975)  is 
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particularly  Interesting.  They  found  evidence  that  the  right 
hemisphere  performs  gestalt,  template-like  operations  on  visual  stimuli 
whereas  the  left  hemisphere  is  responsible  for  more  analytic  operations 
on  visual  stimuli.  In  an  experiment  that  involved  easy  discriminations 
between  a test  face  and  a memorized  face,  they  found  a left  visual 
field  advantage.  A right  visual  field  advantage  was  found  in  a task 
requiring  a difficult  discrimination.  In  both  cases,  the  memorized 
faces  were  learned  under  identical  procedures.  So,  it  does  not  so  much 
seem  that  there  is  differential  storage  of  visual  information  in  the 
right  hemisphere  as  there  is  differential  ability  to  perform  certain 
types  of  operations.  If  so,  the  data  on  hemispheric  asynraetrles  lose 
all  ability  to  discriminate  among  types  of  representation. 

Parsimony  and  Plausibility 

I vfould  like  to  put  together  the  criteria  of  parsimony  and 
plausibility  because  I believe  that  plausibility  is  just  an  extension 
of  parsimony— the  parsimony  of  the  theory  when  integrated  with  our 
general  sense  about  nature  and  hixnan  cognitive  functioning.  These 
criteria  have  some  ^omise  of  discriminating  among  theories  of 
representation.  As  I will  argue,  it  may  prove  possible  to  decide  on 
grounds  of  parsimony  and  plausibility  whether  there  are  two  distinct 
representations,  one  typically  used  for  pictorial  information  and  one 
typically  used  for  verbal  information,  or  whether  there  is  Just  one 
representation  for  all  information.  That  is,  I think  we  may  be  able  to 
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decide  between  a dual-code  model  versus  an  abstract  model.  However,  I 
an  not  optimistic  that  parsimony  and  plausibility  will  be  of  much  help 
in  establishing  the  nature  of  the  representation.  That  is,  even  if  we 
can  decide  in  favor  of  a dual  code  model,  there  seems  little  hope  for 

i deciding  that  one  code  is  verbal  and  one  pictorial.  Similarly,  if  we 

decide  for  an  abstract  code,  I see  little  hope  of  establishing  that  the 
code  is  propositional.  So  perhaps  we  might  be  able  to  establish 
whether  a distinction  should  be  made  between  codes  but  not  be  able  to 
identify  the  character  of  the  internal  codes. 

Propositional  representations  offer  a potential  advantage  in 
terms  of  the  parsimony  of  the  theories  that  are  formulated  in  terms  of 
them.  Because  both  verbal  and  pictorial  information  are  abstracted 
into  a common  representation , it  is  only  necessary  to  propose  one  set 
of  psychological  laws  governing  the  processing  of  that  information. 
This  potential  parsimony  of  propositional  theories  also  offers  a means 
for  their  rejection  on  the  basis  of  an  extensive  research  program.  If 
it  can  be  shown  that  different  laws  govern  the  processing  of 
information  depending  on  its  source  (verbal  or  visual),  this  would  be 
evidence  against  a propositional  theory.  Any  single  piece  of  evidence 
of  this  sort  would  not  be  devastating  for  a propositional  theory.  The 
earlier  theorem  guarantees  that  a propositional  theory  can  always  be 
made  to  account  for  such  perturbations.  For  instance,  one  could 
suppose  that  the  processes  which  operated  on  a propositional 
representation  depended  on  what  was  encoded — verbal  or  visual 
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InfonnatlonJ^  However,  any  extensive  use  of  such  clearly  ad  hoc 
explanations  would  pose  a fatal  strain  on  the  propositional  theory  in 
terns  of  its  plausibility  and  parsimony. 

There  is  some  data  relevant  to  whether  similar  laws  hold  for 
pictorial  and  verbal  information.  The  generation  of  research  that 
attempted  to  show  superior  memory  for  pictorial  information  is  an 
example  of  research  with  this  logic.  However,  as  noted  earlier,  there 
is  now  reason  for  doubting  the  force  of  this  work.  A current  candidate 
which  is  more  promising  is  the  work  of  Cooper  and  others  discussed 
earlier  indicating  that  there  may  be  different  complexity  functions 
describing  matching  of  verbal  and  non-verbal  information.  If  this 
research  were  interpreted  as  Cooper  advocates,  it  would  be  a serious 
strain  on  the  plausibility-parsimony  of  the  propositional  model  with 
its  single  abstract  trace. 

On  the  other  side  of  the  fence,  I can  report  my  work  looking 
for  interference  phenomena  with  visual  vs.  verbal  material  (Anderson  & 
Paulson,  in  press).  Here  evidence  is  found  that  the  same  principles  of 
interference  apply  to  the  two  sets  of  material  (verbal  descriptions 
versus  faces)  and  that  the  absolute  size  of  the  interference  effects 
may  be  quite  similar.  It  is  also  found  that  verbal  and  pictorial 
information  mutually  interfere  with  each  other.  Similar  results  have 
been  reported  by  Pezdek  (1975).  It  is  true  that  there  have  been 
fhequent  reports  of  greater  interference  of  presenting  two  sets  of 
information  in  the  sane  modality  rather  than  different  modalities 
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(Atwood,  1971;  Kroll,  1975;  Salthouse,  1974,  1975;  but  see  Anderson  & 
Bower,  1973)*  However,  these  results  can  be  interpreted  as  resulting 
from  the  greater  similarity  of  the  content  in  the  same-modality 
condition  rather  than  use  of  the  same  modality  per  se.  For  instance,  a 
propositional  encoding  of  two  arbitrary  objects  in  the  visual  modality 
may  overlap  in  such  features  as  color,  shape,  and  size;  whereas,  there 
would  not  be  the  potential  for  such  overlap  between  two  objects  in 
different  modalities.  This  possible  explanation  was  noted  by  Kosslyn  & 
Pomerantz  (1977).  Of  relevance  here  is  the  fact  that  these  r>odality 
effects  are  strongest  over  short  retention  intervals.  Encodings  of 
peripheral  perceptual  properties  of  a stimulus  are  likely  to  be  lost 
rapidly.  Perceptual  properties  provide  the  content  on  which  two 
stimuli  presented  in  the  same  modality  are  most  likely  to  overlap. 

A question  that  has  been  subject  to  considerable  research  is 
whether  visual  and  auditory  information  have  the  sane  short-term 
retention  characteristics  (see  Kroll,  1975,  for  a review).  It  has 
become  apparent  in  this  field  that  subject  strategies,  particularly 
with  respect  to  rehearsal,  are  sufficiently  complex  to  prevent  any 
simple  answer  to  this  question.  I fear  a similar  fate  may  await 
research  on  other  issues.  It  has  been  argued  (e.g.  Newell,  1973)  that 
the  human  cognition  is  so  complex  and  interactive  that  it  not  possible 
to  address  simple  issues  of  the  system,  that  it  is  necessary  to  build 
and  test  complex  models  that  embody  a combination  of  many  assunptions. 
It  is  fear  of  this  possibility  that  causes  me  to  doubt  whether 
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plausibility  and  parsinony  will  yield  any  answer  to  questions  about  the 
mderlying  representation. 

Efficiency  and  Optimality 

It  seems  a reasonable  assuaption  that  the  human  system 
processes  information  in  a way  to  maximize  the  efficiency  and 
optimality  of  its  performance.  Thi  means  that  a constraint  on  any 
theory  is  that  it  not  propose  the  system  is  processing  information 
inefficiently.  It  unfortunately  is  not  always  a trivial  matter  to 
decide  how  the  efficiency  of  the  system  should  be  measured  in  absence 
of  detailed  knowledge  of  its  physiological  implementation.  Anderson  & 
Kline  (1977)  proposed  that  the  efficiency  of  a system  can  be  measured 
in  terms  of  the  efficiency  of  its  computer  simulation.  If  some  such 
proposal  were  accepted,  it  would  be  possible  to  place  considerable 
constraint  on  a theory.  However,  it  is  unlikely  that  there  is  going  to 
be  general  consensus  in  the  field  about  such  a definition  of 
efficiency.  Nonetheless,  it  seems  possible  to  apply  very  general  (non- 
Implementation-speclfic)  notions  of  efficiency  to  impose  some 
constraints  on  theories  of  mental  representation. 

Oie  very  general  consideration  leads  to  an  interesting 
conclusion  about  mental  representations.  It  is  the  case  that  well- 
designed  systems  tend  to  have  special  representations  for  the  kinds  of 
information  they  have  to  process  frequently.  These  representations  are 
designed  to  facilitate  the  kind  of  computations  useful  for  this  kind  of 
information.  For  instance,  we  know  from  physiological  evidence  that 
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visual  and  auditory  information  are  given  very  different  encodings  at 
initial  neural  levels. 

Another  good  example  comes  from  an  advanced  computer  language 
like  INTERLISP  (Teitleraan,  1976).  INTERLISP  has  list  structures  which 
are  useful  for  encoding  symbolic  structures  (like  propositions),  arrays 
which  are  useful  for  encoding  dimensionally  organized  information  (like 
pictures) , and  even  some  string  capabilities  useful  for  encoding 
sequential  information  (like  verbal  input).  The  array  and  string 
capabilities  are  additions  to  the  original  LISP  (McCarthy,  Abrahams, 
Edwards,  Hart  & Levin,  1962)  which  only  had  list  structures.  These 
additions  were  forced  by  the  practical  needs  of  doing  various  types  of 
information  processing  operations  in  LISP.  It  is  the  case  that 
information  encoded  in  arrays  or  strings  can  be  represented  in  lists 
but  at  a severe  cost  to  efficiency  of  processing. 

This  INTERLISP  example  leads  one  inescapably  to  a tri-code 
theory.  That  Is,  it  seems  clear  that  the  hunan  must  process  three 
kinds  of  information — visual-spatial,  verbal-sequential,  and  abstract- 
propositional.  The  INTERLISP  example  reinforces  a belief  that  the 
kinds  of  informatiot.  representations  optimal  for  these  three  domains 
are  different.  So,  it  would  seem  that  there  would  be  a strong  survival 
advantage  pushing  in  the  direction  of  three  separate  codes  with  the 
potential  for  intertranslation  among  them.  Personally,  I find  this  a 
priori  argunent  for  a tri-code  theory  quite  compelling.  It  is 
interesting,  however,  that  it  leads  to  the  opposite  a priori  conclusion 
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than  parsimony.  On  the  a priori  parsimony  criterion  we  gave  the 
advantage  to  a pure  propositional  theory. 

Consideration  of  efficiency  points  in  the  direction  of  useful 
empirical  and  theoretical  studies.  Consider  once  again  the  complexity 
results  reported  by  Cooper.  The  data  reported  would  seem  to  indicate 
that  matching  of  pictorial  material  is  not  affected  by  complexity  but 
that  matching  of  verbal  material  is.  This  is  a study  aimed  right  at 
the  question  of  efficiency  of  operating  on  different  types  of 
representation.  It  is  perfectly  possible  to  mimic  these  results  with  a 
single  propositional  representation  given  the  earlier  formal  argument. 
However,  to  do  so  would  require  proposing  that  when  matching  verbal 
information  the  system  holds  back  on  using  its  potential  for  unlimited 
parallel  processing.  Besides  its  already  mentioned  lack  of  parsimony, 
this  assunption  seems  outrageously  inefficient. 

Even  if  it  were  not  the  case  that  spatial  processing  was 
unlimited  capacity  parallel,  it  seems  possible  to  establish  that  a 
propositional  model  would  have  to  be  made  inefficient  to  mimic  a dual 
code  model  and  a dual  code  model  would  have  to  be  made  inefficient  to 
mimic  a propositional  model.  Suppose  it  were  the  case,  as  it  is  in  the 
INTERLISP  analogy,  certain  computations  could  be  made  much  more  rapidly 
on  spatially  organized  information.  We  could  mimic  these  computations 
with  a propositional  representation  but  this  would  require  proposing 
that  the  propositional  computations  proceeded  much  faster  when 
processing  spatial  information  than  non-spatial  information. 
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Similarly,  we  could  mimic  computations  (e.g.,  abstract  inferences) 
appropriate  for  propositional  operations  with  operations  on  spatial 
representations.  However,  this  might  require  proposing  the  spatial 
operations  occurred  faster  on  abstract  than  spatial  information. 
Either  way,  the  parameters  describing  rate  of  information  processing 
would  not  be  invariant  across  material.  We  would  be  forced  to  propose 
that  sometimes  operations  were  not  performed  as  rapidly  as  they  could. 

These  considerations  lead  to  recomnendations  for  more  research 
of  the  Cooper  variety— where  one  looks  at  processes  on  different 
information,  trying  to  argue  that  the  processes  cannot  be  performed  by 
the  same  mechanisms  proceeding  at  the  same  rate  on  the  same  information 
representations.  Rather  there  are  specialized  representations  with 
specialized  processes  for  special  types  of  information.  Whether  in 
fact  this  will  be  the  conclusion  of  such  research  is  still  up  in  the 
air.  It  may  turn  out  that  there  really  are  ranarkable  invariances  in 
processing  rates  across  types  of  information. 

It  is  worth  noting  that  if  efficiency  considerations  do 
indicate  the  existence  of  multiple  representations,  they  will  do  little 
to  indicate  the  character  of  the  differences  among  the  representations. 
As  noted  in  this  paper's  discussion  of  Kosslyn  & Pomerantz  (p.xxx)  on 
this  point,  we  cannot  decide  whether  a particular  representation  is 
efficient  without  knowing  the  kinds  of  processes  that  will  operate  on 
it.  However,  we  may  be  able  to  use  efficiency  considerations  to  decide 
that  a distinction  among  representations  needs  to  be  made.  So,  the 
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conclusion  on  this  score  is  similar  to  that  fhom  parsimony:  It  may  be 
possible  to  decide  that  there  are  different  representations  without 
deciding  how  they  are  different. 

Conclusion 

To  summarize  the  conclusions  of  the  paper: 

1)  The  picture  metaphor  is  the  only  current  explicit 
interpretation  of  the  image  theory. 

2)  The  frequent  criticisms  made  of  the  picture  metaphor  are  not 
valid.  (Xie  can  have  a viable  dual  code  model  inv'''''»ing  picture  and 
verbal  representat.'.ons  (Fig.  2). 

3)  The  argiments  for  the  necessity  of  a propositional 
representation  are  far  from  compelling.  The  best  have  to  do  with  the 
utility  of  such  a representation  for  inference  making. 

4)  The  arguments  for  imaginal  representations  based  on 
introspections,  computational  considerations,  empirical  results,  and 
physiological  considerations  are  not  convincing. 

5)  It  is  not  possible  to  decide  between  imaginal  and 
propositional  representations  strictly  on  the  basis  of  behavioral  data. 

6)  The  criteria  of  parsimony  and  efficiency  may  allow  a 
decision  about  whether  there  are  different  types  of  internal 
representation  or  just  one  abstract  representation.  This  would  be 
based  on  a research  program  which  investigated  whether  verbal  and 
visual  information  displayed  similar  properties. 

There  are  a number  of  reactions  to  the  possibility  that  we  may 
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not  be  able  to  identify  the  nature  of  the  information  representation. 
A ft-equent  one  among  my  colleagues  is  disbelief  and/or  dismay—this 
would  imply  that  cognitive  psychology  is  not  possible.  However,  I 
think  the  implication  of  this  possibility  is  not  that  cognitive 
psychology  should  be  abandoned  but  rather  that  it  should  undergo  a 
slight  change.  CXir  goal  should  be  to  develop  some  model  capable  of 
accounting  for  hunai  intelligence — that  is,  predicting  behavior  in  a 
wide  variety  of  situations  where  hunan  intelligence  is  manifested.  The 
fact  that  it  may  be  indistinguishable  scientifically  from  other  quite 
different  models  need  not  be  a source  of  unhappiness.  In  fact,  it  is 
possible  to  take  comfort  in  such  equivalences.  If  a particular  model 
is  equivalent  to  many  other  models,  we  can  be  more  confident  in  its 
basic  truth.  Even  if  the  physical  implementation  directly  described  in 
our  model  proves  false  of  the  human  brain,  there  are  many  other  ways 
that  model  could  be  true.  The  possibility  of  equivalent  models  also 
offers  computational  advantages.  Just  as  is  the  case  in  the  wave  and 
particle  models  of  light,  one  version  of  the  model  might  be  useful  for 
certain  computations  and  another  version  of  the  model  for  other 
computations.  It  is  also  worthwhile  to  note  with  respect  to  this 
example  from  physics  that  scientists  were  able  to  make  progress  without 
unique  identifiability. 

The  function  of  science  is  to  discover  what  is  the  case  not  to 
prescribe  what  should  be  the  case.  If  equivalence  and  non-determinacy 
seem  to  be  the  case  we  should  not  be  timid  about  acknowledging  that 
possibility. 
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Footnotes 

1.  Preparation  of  this  paper  was  supported  by  grant  BNS76- 
00959  ft*ora  the  National  Science  Fojjndation  and  contract  N00014-77-C- 
02M2  fhom  the  Office  of  Naval  Research.  I would  like  to  thank  W.K. 
Estes,  S.  Kosslyn,  C.  Lewis,  J.  Perlmutter,  P.  Podgorny,  Z.  Pylyshyn, 
L.  Reder,  R.  Sternberg,  R.  Tourangeau  and  an  anonymous  reviewer  for 
their  connents  on  this  manuscript. 

2.  The  segmentation  is  meaningful  for  myself.  I admit  to 
being  a chess  duffer. 

3.  It  might  be  argued  that  the  representation  in  Figure  2 is 
really  a propositional  representation.  This  could  be  argued  on  one  of 
three  bases:  the  use  of  word  strings  like  "knight  attacks  king",  the 
use  of  the  associative  network  structure,  or  because  of  the  procedures 
Which  we  assune  will  use  this  information.  There  are  two  responses  to 
this  remark.  The  first  is  to  note  that  if  one  agrees  that  this 
representation  is  faithful  to  dual  code  models  and  insists  it  is 
propositional,  one  would  be  conceding  that  there  is  no  difference 
between  dual  code  and  propositional  models.  This  amounts  to  conceding 
the  main  point  that  I will  be  arguing  for  in  the  paper. 

However,  I think  a stronger  response  is  justified  to  this 
argument.  None  of  the  three  bases  for  calling  Figure  2 propositional 
seems  very  good: 

a)  There  has  been  a standard  distinction  in  the  literature 
between  word  strings  and  propositions  which  are  thought  to  be  more 
abstract . 
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b)  Associative  structure  has  never  been  considered  equivalent 
to  a propositional  structure. 

c)  Standard  definitions  of  what  a proposition  is  (abstract,  has 
rules  of  formation,  has  a truth  value)  make  no  reference  to  the 
procedures  that  use  this  information. 

Moreover,  there  seems  to  be  consensus  that  Paivio's  dual  code 
theory  is  not  propositional.  Still,  one  could,  if  he  wanted  to,  extend 
the  notion  of  a propositional  representation  to  Figure  2 without  being 
in  outright  violation  of  any  explicit  definition  of  what  a proposition 
is.  However,  to  do  so  would  seem  to  be  violating  basic  ’’sincerity 
conditions"  on  scientific  discourse.  We  have  to  respect  the 
conventions  of  usage  in  the  scientific  comnunity. 

This  conjecture  about  the  universality  of  propositional 
representation  can  be  seen  as  related  to  a conjecture  known  as  Church’s 
thesis  or  Turing’s  thesis  (Minsky,  1967).  This  is  the  thesis  that  any 
well-specified  behavior  can  be  computed  by  a recursive  function  or 
Turing  Machine.  Given  that  any  well-specified  information  should  be 
capable  of  entering  into  well-specified  computations  and  given  that  the 
recursive  functions  can  be  represented  in  predicate  calculus,  a 
propositional-based  language,  there  is  a sense  in  which  it  would  follow 
from  Church’s  thesis  that  any  well-specified  information  can  be 
propositional ized.  However,  this  argument  implies  a rather 
msatisfactory  way  of  propositionalizing  information.  Therefore,  I 
think  it  is  better  to  let  this  conjecture  stand  as  an  independent 
claim.  It  is  explicitly  accepted  by  Kosslyn  & Pomerantz  (1977). 
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5.  For  instance,  if  a person  reports  to  us,  "My  mind  is 
swarming  with  ideas,"  would  we  want  to  assune  a mental  representation 
of  a swarm  of  bees? 

6.  Ihe  hedge  "virtually"  is  inserted  because  we  can  assune 
that,  whatever  its  representation,  there  are  certain  behaviors  of  which 
a system  is  logically  incapable — for  instance,  solving  the  halting 
problem  (see  Minsky,  1967).  Note  these  "logical"  predictions  about 
behavior  would  be  true  of  all  representations  and  so  do  not  provide  a 
way  of  discriminating  among  them. 

7.  The  class  of  primitive  recursive  functions  is  very  large 
but  is  a subset  of  the  recursive  functions  which  can  be  computed  on  a 
general  purpose  computing  device  equivalent  to  a Tbring  machine.  One 
way  of  thinking  of  the  difference  between  primitive  recursive  functions 
and  all  computable  functions  is  that  it  is  possible  to  place  time 
bouids  on  the  amount  of  computation  required  to  perform  a primitive 
recursive  function  whereas  such  time  bounds  do  not  always  exist  for 
more  general  recursive  functions.  The  class  of  primitive  recursive 
functions  is  quite  powerful,  nonetheless.  It  is  difficult  to  imagine 
that  any  process  computed  by  the  brain  is  more  powerful  than  primitive 
recursive.  So  under  very  minimal  assumptions  about  its  nature,  f will 
have  a computable  inverse. 

8.  If  E or  E map  more  than  one  input  into  the  same  value  I or 
I the  Inverse  of  E or  E*  can  map  I or  I*  into  any  of  these  inputs. 
This  use  of  inverse  may  not  be  totally  conventional  but  is  what  is 
needed  for  the  argixnent. 
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9.  The  point  of  this  formal  analysis  was  to  show  that  one 
cannot  decide  issues  of  representation  in  the  abstract,  that  one  must 
also  specify  the  processes  that  operate  on  the  representation.  It 
might  be  asked  whether  this  point  can  be  turned  around.  Is  it  possible 
to  decide  issues  of  process  in  the  abstract?  Could  we  get  two  very 
different  processes  to  mimic  each  other  by  choice  of  structure  in  the 
way  it  was  shown  that  two  structures  could  mimic  each  other  by  choice 
of  process?  The  situation  is  not  exactly  syinnetric  in  that  it  is  not 
possible  to  assign  a structure  for  a process  in  the  way  it  was  possible 
to  assign  a process  for  a structure.  A process  must  specify  the 
structuTE  of  its  input  and  of  its  output.  Nonetheless,  there  probably 
is  considerable  indeterminacy  in  process  assunptions.  We  can  have  two 
processes  mimic  each  other  by  changing  the  processes  with  which  they 
interact,  tor  instance,  we  can  get  different  processes  T and  T*  to 
mimic  each  other  by  changing  the  encoding  processes,  E and  E*,  and  the 
decoding  processes,  D and  D*.  Thus,  it  may  not  be  possible  to  test  one 
process  without  specifying  the  processes  with  which  it  interacts. 

10.  This  can  be  a fairly  plausible  assumption  for  certain 
processing  formalisms  such  as  production  systems  (e.g.,  Anderson, 
1976).  A production  cannot  apply  unless  the  content  of  the  information 
meets  certain  specifications.  Thus,  we  might  have  certain  types  of 
productions  that  only  applied  to  propositional  representations  that  had 
verbal  content  or  to  propositional  representations  that  had  pictorial 
content. 
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Table  1 

A Propositional  Ehcoding  of  the  Letter 
In  Figure  4 


C0MP0SE(R,S1,S2,S3,S4,S5,S6) 
STRAIGHT(S1,P1,P2) 
311^1(211(32,  PI,  P3) 
STRAIGHT(S3,P3,P4) 
STRAI(21T(S4,P3,P5) 
STRAI(21T(S5,P4,P6) 
HALF-CIRCLE (36 , P2 , P4 ) 
LEN(rrH(31,10) 

LENGTH(32,20) 

LENCTH(33,10) 

LENGTH(S4,20) 

LEN(mi(S5,23) 

LEH(?rH(36,31) 


A1«LE(S1,32,90) 

AIIQ:^(32,33,90) 

AN(a^(32,34,180) 

AIK1E(S3,34,90) 

AMCLE(33,35,117) 

AXIS(R,P3) 

VECT0R-AN(2.E(P1,90) 

VECT0R-ANGLE(P2,63) 

VECT0R-ANC}LE(P4,0) 

VECTOR  ANaE(P5,270) 

VECTOR- ANGLE(P6, 315) 


Figure  Captions 
A end  game  chess  position. 

A dual  code  representation  of  the  chess  board 
in  Figure  1. 

(a)  A representation  of  the  transformation 
of  information  representations  in  model 

H. 

(b)  A representation  of  the  transformation 
ofjinformation  representations  in  model 

M which  mimics  M. 


Figure  1 
Figure  2 


Figure  3 


The  letter  R whose  propositional 
coding  appears  in  Table  1. 
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